1. Introduction

The value of a weather forecast to a user is about much more than accuracy. The most important aspect of the design of a forecast delivery system is to optimize the flow of the opinion of the forecaster directly to the user. The user should then utilize this opinion in an optimal way, given the relative costs of occurrence of the event and protection from the event. In this paper it is assumed that the user, in this case an airline, would want to use the forecast in a way that optimizes the economic outcome without compromising safety.

This issue of obtaining the maximum value from the expertise of the forecaster is the main topic of this study. Meteorological authorities have gone to great lengths to measure weather forecast accuracy, but less focus has been given to forecast value. Many papers have been written on the subject of accuracy and value in weather forecasts (e.g., Murphy 1977, 1985; Katz and Murphy 1997). These describe the extra value inherent in probabilistic forecasts. More recently studies have been published on the value of ensemble numerical forecasting and its relative value with respect to increasing the resolution of models. Wilks (2001), Zhu et. al. (2002), and Richardson (2000) use broadly similar methodology in ascribing the value of probabilistic forecasts based on, inter alia, numerical ensembles, with respect to climatology and perfect forecasts. Of interest for the current study, Wilks found that forecasts exhibiting consistent overforecasting (i.e., a low decision threshold) produce greater value for situations with low cost–loss (C–L) ratios. This is shown to be the case in this study for airline costs. Most short-duration flights have very low C–L ratios. Wilks also found that the use of probabilistic methods achieved the greatest increase in value over climatology, with respect to perfect forecasts, for forecast probability densities typical of short-range forecasts. Zhu et al. use a relative operating characteristic (ROC) area summary measure to demonstrate the value of the multiple decision thresholds inherent in probabilistic ensemble methods.

However, little or no attention to the issue of value seems to have occurred in the field of aviation forecasting. Given the large financial impact that weather forecasts have on airline operations, particularly terminal aerodrome forecasts (TAFs), the topic would seem to deserve some recognition.

The results of this paper will show that considerable potential value is lost by the traditional method of providing weather information in TAFs in categorical form, that is, as a binary, yes–no product. The current rules do allow forecasters to use probabilities, for some elements like thunderstorms and fog. For example they can say “PROB30” for the occurrence, which means a 30% chance of occurrence. However, airlines are obliged by regulations to carry the full fuel requirement equivalent to a forecast of 100% confidence. So the use of the PROB30 is redundant, and the forecast is effectively completely categorical. Forecasters know this, and also have some concept of the consequences of missed events. It will be shown that they adopt quite varying tactics, demonstrating a variety of attitudes to these consequences. Forecasters are also able to use temporal categorical variations to mean conditions when formulating TAFs. If the mean conditions are above the special lowest alternate minimum (SLAM), the term INTER (TEMPO) refers to periods of less than 30 (up to 60) min below the SLAM. It is shown below that use of these modifiers can be interpreted as crude probabilistic forecasting.

2. Experiment design

Forecasters at three Australian forecasting offices were asked to estimate their confidence, to the nearest 10%, that the weather at five different lead times will be below the SLAM for a particular aerodrome. The SLAM comprises values of cloud base and visibility, as well as weather type, and is the level used to determine fuel carriage for most passenger-carrying aircraft. The lead times are 1, 3, 6, 12, and 18 h. Forecasters formulated the probabilities at the same time they produced the routine issue of the TAF. Only routine issues of the TAF were tested. This was done so that the lead time–skill relationship was not skewed. Nonroutine amendments are usually issued to amend the TAF at short lead times, and so counting these would probably bias the skill in favor of the shorter lead times.

One purpose of the experiment was to demonstrate the use of the signal detection theory (SDT) model of forecasts for TAFs, and to ascertain whether the data can be fitted to the Gaussian model as used by Mason (1982). Another aim was to investigate differences in the forecast tactics between individual forecasters, and to demonstrate the effects of these differences on the financial outcome to airlines.

Other studies, most notably Mason (1982), have shown that probabilistic forecasts of elements like rain, storms, and temperature closely fit the SDT model. But in order to use Mason's cost–value model (I. M. Mason 2001, personal communication and described below), it was necessary to confirm that forecasters' probability estimates of poor weather at airports fit the Gaussian assumptions of the SDT model.

The trial data were provided by forecasters at the Victorian Regional Forecasting Centre (Vic RFC), the Sydney Airport Meteorological Unit (SAMU), and the Townsville Meteorological Office (TVL). All forecasters were volunteers. Data have been accumulated at Townsville since December 1999, at Vic RFC since March 2000, and at SAMU from about April 2000. Data have been analyzed up to September 2001.

3. Theory

a. Signal detection theory

The primary advantage of SDT is its ability to model the effect of forecasters' decision thresholds on accuracy. It provides a measure of a forecaster's acuity, or ability to discriminate between two signals, one just noise, and one noise plus signal. For a treatment of SDT as applicable to weather, the reader is referred to Mason (1982). Other interesting applications of SDT in meteorology are described in Harvey et al. (1992) and Levi (1985). Swets (1996) gives a full treatment of SDT in general, with most applications being in the medical and psychological fields.

The linchpin of SDT is the assumption that, prior to a decision, there are two overlapping probability distributions: the weight of evidence for the event occurring, and weight of evidence against the event occurring. This is illustrated in Fig. 1, with f₁(x) representing the evidence for the event, and f₀(x) the evidence against. The y axis is the weight of evidence, the x axis shows values of χ, the decision threshold. The area under the probability distribution for f₁(x) to the right of χ_c, the critical decision threshold, is the probability of a hit (hit rate). Similarly, the area under the f₀(x) curve to the right of χ_c is the probability of a false alarm (false alarm rate). If the forecaster becomes more conservative (i.e., his critical decision threshold χ_c decreases or moves to the left), both the hit rate and false alarm rate increase. Conversely, if the forecaster becomes more adventurous (i.e., less inclined to forecast the event), the false alarm rate decreases, but so too does the hit rate. Note that the separation of the means of the two distributions is denoted as d'. This parameter can be used as an index of skill as it defines the intrinsic ability to discriminate between the two distributions.

The form of the distributions in Fig. 1 is assumed to be Gaussian. Studies of human decision making under conditions that produce uncertainty have demonstrated Gaussian behavior. Mason (1982) showed this for weather forecasts. This behavior is demonstrated for the data in this experiment below. The Gaussian assumption is also convenient in that the calculations fall out nicely.

The formal definition of hit rate, h, is Pr(Forecast = Yes | Event = Yes), that is, the probability that the event is forecast given that it happens. This is the same as the probability of detection (POD) as widely used in meteorology. Similarly false alarm rate, f, is Pr(Forecast = Yes | Event = No), that is, the probability that the event is forecast given that the event does not happen. Note that f is different from FAR, the false alarm ratio, a term encountered in meteorology and often confused with f. FAR in fact is Pr(Event = No | Forecast = Yes).

In the trial, forecasters were asked to indicate their confidence of below SLAM weather at each lead time to the nearest 10%, so that there are 11 different decision thresholds from 0% to 100%. The hit rate and false alarm rate can then be calculated for forecast probability greater than or equal to each of the 11 different decision thresholds. These are then plotted against one another to produce a ROC. By way of example, the ROCs produced in this experiment are shown later (Figs. 5–7).

The diagonal on the ROC is the line h = f. On the diagonal a forecaster has the same chance of a hit as a false alarm, and so the diagonal is defined as zero skill. A perfect forecast is indicated by a ROC from (0,0) to (0,1) to (1,1). So a reasonable skill measure is the area under the ROC curve. Mason (1982) calls this A_z once the Gaussian model is applied. Here, A_z has been shown to be independent of the climatological rate of occurrence (e.g., Mason 1989) and also, most importantly, independent of the decision criterion χ_c (e.g., Harvey et al. 1992). These are advantages of the SDT summary measure A_z. The dependence of some traditional skill scores on decision thresholds is described in Swets (1986, 1996), Mason (1989), and Harvey et al. (1992).

As mentioned in Harvey et al. (1992), A_z is a measure of potential forecast performance. As is demonstrated in their paper, and in this study, the economic value of the forecast depends on the decision criterion, χ_c, as well as on the accuracy. As a measure of skill, A_z qualifies as a strictly proper score. This means that it is not possible to optimize the score by hedging. Because conventional skill scores are, to varying degrees, dependent on decision threshold, a shrewd forecaster, after a little research and experimentation, could optimize his or her score by adopting a decision threshold that optimizes the particular score by which he or she is being assessed.

Returning to the Gaussian distribution of the signal detection model in Fig. 1, h and f can be expressed in terms of the location of the decision threshold, χ_c, on the x axis of the overlapping normal distributions. A more expansive treatment of this can be seen in McMillan and Creelman (1991), Swets (1996), or Mason (1982).

The likelihood ratio is defined as β = f₁(x_c)/f₀(x_c). If the mean of f₀ is set to zero, the mean of f₁ becomes d′, the separation of the means. If the variances of the two distributions are assumed to be equal, one can connect χ_c and d′ as follows. From the formula of the normal distribution,

View Expanded

From Mason (1982), the likelihood ratio β can be expressed through Bayes's theorem as a ratio of the odds of the probabilities of the event at χ = χ_c, p, to the odds of the climatological probability, p_c:

βppp_cp_c

A link is thus available from forecast probability to hit rates and false alarm rates, using as variables p_c and d′. Again assuming equal variances of the two distributions, d′ can readily be calculated from the data in the standard 2 × 2 contingency table of Table 1. Here, d′ is just the difference between the normal deviates of h and f. As h is an estimate of Pr(Forecast = Yes | Event = Yes), h = d /(d + b), and similarly f = c/(c + a). The normal deviates of these can be calculated and subtracted to yield d′. The degree of validity of the assumption of equal variances in the context of this paper will be discussed below.

b. Forecast value

In any forced choice, binary outcome (yes–no) forecast situation, the outcome can be summarized by the traditional 2 × 2 contingency, for example, Table 1. The outcomes, for the sake of intuitive understanding, are described as true positives (hits), true negatives (correct rejections), false negatives (misses), and false positives (false alarms).

Harvey et al. (1992) developed a relationship for expected value of a forecast, using the four conditional probabilities from the contingency table, and the value of each of the outcomes. They arrive at a relationship for the expected value (EV) of a forecast. Substituting terminology used in this paper for their terminology, the relationship is

View Expanded

where p_c is the climatological rate of occurrence of the event, or Pr(Event = Yes), the Bayesian prior probability; h is hit rate; f is false alarm rate; and V_TN is the value of a true negative, V_FN the value of a false negative, V_FP the value of a false positive, and V_TP the value of a true positive. Both V_FN and V_FP have negative values and are better called costs.

Equation (1) was derived independently by Mason (2001, personal communication). He uses the fact that perfect forecasts have h = 1 and f = 0, and so he simplified (1) to provide an expression for the expected cost of an imperfect forecast with respect to a perfect forecast:

View Expanded

Mason defines (V_TN − V_FP) as the false alarm cost, the cost of incorrectly forecasting an event, and (V_TP − V_FN) as the miss cost, the cost of not forecasting an event. Making this substitution,

View Expanded

Note that by definition, miss cost does not include any costs already incurred by a hit, or correct forecast of an event. In the context of a TAF, an aircraft may not be able to land regardless of whether the bad weather was or was not forecast. The miss cost is only that cost attributable to the event not being forecast over and above the cost accrued if it had been correctly forecast. False alarm cost is easier to ascribe, as there is no weather-related cost attributable to a correct rejection, and it is thus the cost of protective action. In the TAF situation, this would be the cost to carry extra fuel and perhaps lost payload.

Mason (2001, personal communication) has shown that if (2) is differentiated with respect to p, the forecast probability, and the result set equal to 0, the value of p(opt), the value of the optimal forecast probability that minimizes the expected cost, is defined thus:

p(3)

where CR is the cost ratio and equals the false alarm cost divided by the miss cost.

Mason's derivation of (3) goes as follows: differentiating (2) with respect to p and setting the result to zero gives

View Expanded

It can been shown (e.g., Green and Swets 1974) that the slope of the ROC, dh/df, is the same as the likelihood ratio, β, at the corresponding value of χ_c. Substituting [β/(p/(1 − p)] for (1 − p_c)/p_c gives

pp

So, as one would logically expect, the optimum decision threshold is a function of the costs of the outcomes only. For a particular flight, the magnitude of the expected cost will be determined by the skill d′ and p_c, but for a given d′ and p_c the minimum value of expected cost will be at p = p(opt).

4. Results and discussion

5. Cost analysis

Costs were provided by Qantas Airways for a flight from Singapore to Melbourne. The actual figures are commercially confidential, but the methodology for calculating the false alarm cost and miss cost is described using symbols for the dollar amounts.

When the flight plan is done prior to the flight leaving Singapore, the Melbourne TAF forecasts either alternate (ALT) conditions, INTER or TEMPO deteriorations in the weather, or good conditions above the alternate minima. For the sake of the current exercise, INTER and TEMPO forecasts are ignored. INTER forecasts (30 min of holding fuel) are absorbed into the company's reserve, and at Melbourne TEMPO is forecast only rarely. If the Melbourne TAF forecasts below alternate conditions, the aircraft takes on sufficient fuel to make an approach into Melbourne, and then to fly back to Adelaide if it cannot land at Melbourne. Adelaide is near the track about 300 mi NW of Melbourne. In this case the pilot will usually make an approach into Melbourne, knowing he or she has enough fuel to abort and fly back to Adelaide. If the Melbourne TAF forecasts conditions above the alternate minima, the pilot usually flies on to Melbourne and lands. However if the weather at Melbourne is below the alternate minima when the flight reaches Tailem Bend (TBD), a point on track abeam Adelaide, the pilot will divert to Adelaide.

The costs required in order to calculate the false alarm cost and miss cost are shown in Table 2. The false alarm cost is readily available from this information, being the “cost to carry” the extra diversion fuel of $C. Calculation of the miss cost is more complex. Remember that the miss cost is that cost caused by a diversion over and above that cost accrued if the below minimum weather was correctly forecast, that is, above the cost of a hit. The cost of a hit depends on whether the pilot can actually land at Melbourne when the weather is below the alternate minimum. If the pilot can land safely at Melbourne when the forecast is a hit, then there is no extra cost over the false alarm cost of $C. However if he cannot land, then the aircraft must fly to Adelaide, and return to Melbourne when able to land there. The cost of the diversion is $(C + H + E + F) minus $I for fuel that is not used. Then the question arises: what proportion of the time that the weather is below the alternate minimum can the pilot still land?

Every airport has a level below the alternate minimum to which a pilot may descend, at which time he or she aborts if visual reference of the airstrip has not been attained. This level is called the instrument landing system (ILS) minimum. Thirty years of synoptic observations were analyzed for the Melbourne airport. It was found that of the number of occasions that the alternate minimum was breached, 28% of these also breached the ILS minimum, thus precluding landing. So the calculation of hit cost proceeds thus:

View Expanded

Inputting actual dollar values,

View Expanded

From the Bureau of Meteorology's TAF verification, a typical value of the (h, f) pair is (0.75, 0.12) for Vic RFC forecasters at a lead time of 6 h. Using the assumption of equal variances, p_c = 0.02 and d′ = 2.1 (typical skill measured at Vic RFC in the experiment at 6- and 12-h lead time), the closest value of the forecast probability that produces (h, f) values nearest (0.75, 0.12) is about 0.02. So one can say that the forecasters at Vic RFC are operating with an effective average probability of about 0.02 as their decision threshold for this forecast.

Refer back now to (2) and use the false alarm cost and miss cost calculated for this flight. Figure 15 is a plot of cost versus decision threshold for this flight. At a decision threshold probability of 0.02, the cost of the uncertainty in this forecasts is $231. If the forecast was reliably made at the optimum decision threshold, 0.117, the cost would have been $128. So a perfectly reliable forecast of the probability of below alternate minimum conditions would, in the long run and at the same skill, save about $103 per flight for the Singapore to Melbourne route. This is about 45% of the total cost of the uncertainty of the forecast.

Consider now the reliability diagrams for Vic RFC in Fig. 13. Assume a forecaster at Vic RFC forecast at a decision threshold of 0.117 for the Singapore to Melbourne flight. From the reliability diagrams, he or she would expect roughly 0.07 as the frequency of occurrence in the long run. This can then be considered as the effective decision threshold. A decision threshold of 0.07 leads to a cost of $135. So for this flight, even the moderately reliable forecasts as currently produced would provide most of the savings (41%) gained by perfectly reliable TAFs (45%).

Figure 16 shows, for the same flight, how the modeled cost of the forecast varies with d′, the index of skill. The two graphs are for the optimum decision probability of 0.117, and for the estimated actual decision threshold 0.02. The large difference in cost outcome for the two different decision probabilities is obvious, especially at low skill. Note that if the decision threshold is optimal, a decrease in skill does not matter all that much in economic terms. Therefore one could suggest that using near-optimal decision thresholds is more important than increasing skill, especially for low skill forecasts. The optimal decision threshold is solely a function of the operating costs of a particular flight and, thus, varies between flights. So reliable estimation of the probability of occurrence of the event must be applied to each flight in order to optimize savings, and the cost parameters for each flight used to determine whether extra fuel is required.

Note that the method of calculating the miss cost uses a quite crude climatology. The factor of 0.28, being the ratio of the time conditions are below the ILS minimum to that below the alternate minimum, is for all times of the day and months. A superior value for the miss cost, and thus CR and p(opt), could be obtained if there were adequate data to determine a matrix of these factors for different times and months. This would undoubtedly increase the potential for savings by using probabilities. Furthermore, if the conditional probability of weather below the ILS minimum given that the weather is below the alternate minimum could be forecast with a skill better than climatology, a more accurate p(opt) value could, on average, be calculated on a case-by-case basis.

Figures 17 and 18 are the same two graphs for a shorter flight of duration about 2 h, between Brisbane and Townsville. The false alarm cost is $200, the miss cost $4800, p_c = 0.01, and the measured d′ for Townsville forecasters was 2.0 at 3-h lead time. The measured average decision threshold for the group was 0.015, and taking into account the reliability diagram, the “effective” threshold would be about 0.01, even more conservative than that for Melbourne forecasters. At this threshold, the cost of the errors in this forecast is $38, and at the optimal decision threshold of 0.04 the cost is $27. So, in relative terms, the savings accrued by the use of probabilities is less for the shorter flight than for the longer, international flight. This is undoubtedly because of the relatively high false alarm cost, that is, the cost of carrying fuel over a long distance unnecessarily. Miss costs involve significant components that are fixed and do not depend on route length. Another interesting comparison between the two flights is the effect of a more adventurous (higher) decision threshold. The cost rises much more quickly from the minimum for the shorter flight as the decision threshold rises than for the long flight. In Fig. 17, if the decision threshold rises above 0.25, the cost savings accrued by using probabilities are lost. So the “sweet spot” for savings using probabilities is less than for the long flight.

6. Forecast system design

Meteorologists are judged on the outcome of forecasts, so it is reasonable to assume that they want to achieve the highest possible accuracy. On the other hand the desired outcome for commercial users of the forecast is to use the forecast to optimize the outcome of their particular enterprise.

Brown and Murphy (1987) studied the particular case of fire weather forecasts of credible interval temperature ranges, relative humidity, and wind speed. They found bias in the forecast of fire weather elements toward values that produced more caution, that is higher potential fire rate of spread. At higher wind speeds and lower relative humidities, forecasters are extremely cautious with their probability estimates. To quote from the abstract of that paper: “The forecasts also exhibit modest but consistent biases which suggest that the forecasters are influenced by the impacts of the relevant events on fire behavior. These results underscore the need for probabilistic fire-weather forecasts.”

On the other hand, Winkler and Murphy (1979) and Murphy et al. (1989) looked at the reliability of credible interval temperature forecasts. Murphy and Winkler (1992), Hamill and Wilks (1995), and Murphy and Winkler (1976) studied probability of precipitation and temperature interval probability forecasts. A common element in all these studies was that there were no apparent safety or economic consequences to the estimates of probability. In all these studies the estimates were quite reliable; that is, the observed frequency of events closely resembled the forecast probability.

So it seems that forecasters can generally formulate reliable estimates of probability of occurrence when the outcomes do not have known consequences that affect the decision. On the other hand, if the outcomes have a safety connotation, or if a particular outcome can impact adversely on the forecaster, considerable overforecasting can occur. It is interesting that forecasters rarely, in fact never to the author's knowledge, have to write a report on a forecast error due to a false alarm on a TAF. However a missed event often triggers a “please explain.”

Much has been written in the meteorological literature about probabilistic forecasting, in particular about its inherent value. Murphy (1977) discussed the relative value of climatological, categorical, probabilistic, and perfect forecasts. Murphy expressed forecast value as a function of the cost–loss ratio. He demonstrated that the value of the probabilistic forecasts was the same as the categorical forecasts when the cutoff probability used to define the categorical forecasts was about equal to the cost–loss ratio. At other values of the cost–loss ratio, the value of the probabilistic forecasts was greater. Mason (2001, personal communication) pointed out that the ratio CR/(1 + CR) is equivalent to the traditional cost–loss ratio. Returning to the earlier nomenclature for forecast value, V_TN = 0, V_FN = L, V_FP = C, and V_TP = C, where L is the loss due to an event and C is the cost of protection. By definition CR = (0 − C)/(C − L), and the optimal decision threshold, p(opt) = CR/(1 + CR) = C/L, the cost–loss ratio. So the decision threshold that maximizes economic value is seen to be the same in the signal detection environment as that determined originally by authors in earlier papers.

Over the years there have been some attempts to cope with uncertainty in the TAF. Forecasters can specify in a TAF the probability of occurrence of fog and thunderstorms, these being elements that will probably preclude landing. The problem with this methodology is that the probability is ignored operationally, and legally airlines are bound to carry the extra fuel irrespective of the probability estimate. Several years ago, in order to correct what was perceived as excessive fuel usage in the United States and Europe where there are many suitable alternates, the International Civil Aviation Organization changed the rules and forecasters were instructed to mention only probabilities of occurrence of 30% and above on TAFs. In other words, the system was changed in a way that attempted to raise forecasters' decision threshold. In Australia, for example, many cautious forecasters in tropical areas were often forecasting 10% chance of thunderstorms, which was often a realistic estimate given the isolated and sporadic nature of these thunderstorms. When the instructions were changed, most forecasters just changed 10% to 30% (personal communication and experience!), because the perceived consequences had not changed.

As this change naturally tended to cause more misses, Qantas and the Bureau of Meteorology initiated the Code Grey system. This forecast is issued for major Australian airports during the afternoon, valid overnight and the next morning. Its purpose is to alert airlines to even the slightest chance of fog, even if this is only 1%–2%. As the miss cost for Quantas' international flights is high, they then carry fuel over and above that required by the categorical TAF. So Qantas realizes the value in the knowledge of a small probability of a significant event. If forecasters really operated on a decision threshold of over 0.30, the results would obviously be highly suboptimal. But this is what “the system” expects of them, and so they are caught in something of a dilemna.

It can readily be seen that the current TAF system is the result of a legalistic evolution to a system of categorical forecasting. The system takes no account of the probabilistic nature of forecasters' thought processes, in particular the variable decision threshold that is driven by consequences and individual forecaster's attitudes to those consequences.

7. Conclusions

Considerable economic benefit through lower fuel usage is potentially available to airlines if TAFs were expressed as estimated probabilities of breaching the alternate minimum. This would enable airlines to unlock the value in forecasters' ability to provide reasonably reliable estimates of the probability of occurrence of these events. The amount of benefit would depend on three main factors:

1. The ability of airlines to accurately specify the miss cost and false alarm cost for every flight. The decision process for the two flights given as examples is a straightforward and simple example. Many flights have multiple possible alternate destinations and PNR decision points. An example of the possible complications is a flight from Tokyo to Perth, Australia, operated by Qantas with a Boeing 767 (B767). This flight is near the limit of endurance for a B767. If the Perth TAF has an alternate requirement and the flight is carrying a full payload, the pilot must land at Darwin to take on extra fuel. So the false alarm cost increases dramatically due to payload considerations. The effect is to raise p(opt) from 0.02 to around 0.20, an order of magnitude greater. While many flights are simple, single-point decisions like the two examples, others are more complex, and considerable effort would be required by airlines to specify the full suite of costs.

2. The willingness of regulators to allow airlines to plan fuel requirements from a probabilistic forecast. For this to happen, it would be necessary to prove that probabilistic TAFs provide a commercial benefit to airlines and do not impact negatively on safety. To this end, a trial is commencing, in conjunction with Qantas and American Airlines. Costs will be developed for several flights from each airline. About an hour before departure, forecasters will estimate the probability of weather being below the alternate minimum at arrival time. A comparison will be made of the economic outcomes of this method of planning fuel and the traditional categorical TAF.

3. The ability of forecasters to achieve reliability with their probability estimates. The study in Murphy and Daan (1984) shows that forecasters can learn and that their probability estimates can become more reliable. The results of their experiment suggest that even better changes in reliability could be obtained with more frequent feedback to forecasters.