1. Introduction

In view of the rising angst over budget cutting, we believe it to be critically important to determine quantitatively the economic value of weather forecasting services. In the past few years, the Atmospheric Environment Program in Canada, the Bureau of Meteorology in Australia, the U.K. Meteorological Office, and the Meteorology Service of New Zealand have, among others, experienced unprecedented upheavals in their public forecasting services. Changes such as the effort to “privatize” and to “self-finance” forecasting services have been motivated mostly by budget-reduction drives in their respective federal governments. Although the National Weather Service (NWS) in the United States is also undergoing a major change by implementing its Modernization and Associated Restructuring (MAR) program, to date the impacts on the NWS have remained modest in terms of privatization, at least in comparison to those just mentioned. It remains to be seen how long that relatively small impact will continue.

Draconian budget cutting is, effectively, an externally imposed constraint on public weather services. Outside of the United States, slashing of the public weather service budget has some of its origins in the inability of public weather service management officials to offer a quantitative estimate of the value of its public forecasting service. In at least one case (which is unidentified to protect the persons who made us aware of it), national-level public weather service officials were asked by policy-makers about the effects of proposed budget cuts and could not produce quantitative documentation of the potential negative impacts of the proposed reductions. This made their service an easy target in the political arena; efforts to balance national budgets eventually will force public weather services in all nations to respond to this sort of question. As it now stands, the United States is poorly equipped to make intelligent decisions in the current and future budget situations. We view the inevitability of the inquiry everywhere, including the United States, to be reason enough to try to avert a possible negative outcome of a drastically reduced public forecasting service. In fact, this very issue has already come up in congressional hearings.¹

We also should be prepared to admit that the outcome of such an analysis might enhance the chances for a drastically reduced public sector forecasting service. After all, there is a possibility that the study could reveal that it would be more effective to “privatize” large segments of current NWS services (see Brooks et al. 1996, hereinafter BFD96). To mandate in advance the outcome of a quantitative analysis would be unscientific, however. We prefer that when the question is asked, quantitative documentation of forecast value is readily available, to prevent a potentially disastrous impact on the public forecast infrastructure. If it becomes clear afterward that such cuts have been a mistake, it will be a long, expensive process to reconstruct that infrastructure.

We note that, although our emphasis is going to be on economic value, there is a critical aspect of forecast value we are not considering, namely, that associated with reducing human casualties associated with hazardous weather. It is certain that casualty mitigation can have a substantial economic impact in its own right, but we are not prepared to consider such issues as how society values human life or the economic impact of injuries and fatalities. Budget pressure is essentially an economic fact, and so we are emphasizing the importance of assessing the economic value of weather forecasts. Decision-makers need information about economic factors in forecasting, as well as information about the efforts to reduce casualties.

We are not suggesting that casualty mitigation is irrelevant or unimportant. In fact, in BFD96, we recommend an emphasis on hazardous weather in the public weather services. However, we (like most meteorologists) are not well versed in the literature and techniques of assessing the economic value of forecasting in terms of casualty reduction (see Fisher et al. 1989 for a summary). We believe, further, that the purely economic value of forecasts is sufficiently high that maintaining public weather services makes sense in financial terms, even disregarding contributions to public safety (and the casualty-related economics). The fact that we believe this might not carry much weight in public policy-making, however; what is needed is hard, quantitative information about economic value.

In this essay, we propose a framework for establishing the value of weather forecasting services in section 2. In section 3, we will give some examples of innovative approaches to deriving answers to the question, “How valuable are these weather services?” Finally, in section 4, we offer some discussion of how this framework could be applied within the forecasting community of the United States.

2. Forecast value versus accuracy

As Murphy and Ehrendorfer (1987) and Roebber and Bosart (1996) have enunciated so clearly, there is considerable difference between forecast value and forecast accuracy. The relationship between them is not so simple as intuition might suggest. As a simple counterexample to this initial intuitive belief, consider a forecast service that is always wrong: whenever the forecast is for rain, it is dry; whenever the forecast is for warming, it gets colder; whenever the forecast calls for sun, it is cloudy. Clearly, such a forecast service is terribly inaccurate, but a reasonably intelligent user would find it quite valuable. All this intelligent user would have to do is prepare for the opposite of what the forecast calls for and then reap whatever benefits accrue. This example is rather contrived; however, as we shall explain, it is indeed possible for accuracy to increase and yet value to decline in less contrived circumstances.

A factor that makes the connection between accuracy and value so difficult to define is the cost/loss ratio.² That is, if the user of a forecast takes some action in response to the forecast, that action has a cost, C. If the user fails to take that action, however, there may be a loss, L, associated with that failure to act. A simple example is a user growing crops that are sensitive to freezing. There are actions that the user can take (e.g., spraying fruit trees with water) to diminish the threat of freezing weather. These actions have a cost that a grower would not want to incur needlessly. However, failing to take those actions in a freeze means some amount of crop loss, creating a proportionate loss of income. Every user of weather information has a cost/loss ratio, (C/L), and, generally speaking, that ratio differs for each user. Some users are not knowledgeable about their cost/loss ratio and so are handicapped in determining whether to take a protective action.³

Even when C/L is known, however, Murphy and Ehrendorfer (1987) have noted that it is still difficult to be precise about the relationship between accuracy and value. They point out that it typically is possible to obtain a single-valued relationship between accuracy and quality only when making a number of simplifying assumptions about the problem. Of particular import is the process by which forecast accuracy is specified; generally, this is not completely determined by single scalar measures of accuracy (e.g., a Brier score or a skill score).

It is through the choice of an accuracy measure that the counterintuitive result of value declining as accuracy increases can arise. Consider another simple example, this time for dichotomous (i.e., yes/no) forecasts of a dichotomous event. This problem can be described in terms of a simple 2 × 2 contingency table (Table 1). One accuracy measure is the probability of detection [POD, defined as x/(x + y)]. Another is the false alarm ratio (FAR, defined as z/(x + z)]. If the POD is selected as an accuracy measure, then a user whose value depends on FAR might well find value declining as POD increases, since POD can be increased simply by predicting the event more frequently, without regard for the FAR. As noted by Murphy and Ehrendorfer, no single accuracy measure can describe all aspects of accuracy,⁴ even for this simple 2 × 2 case. For the same reason, no accuracy measure can be appropriate for all users. Thus, a comprehensive theoretical expression for the relationship between accuracy and value can be hard to find. Value is a complex, user-dependent function and although purely prescriptive studies (Murphy 1994) should continue, they cannot provide definitive answers.

Since statistical theory offers only limited insight into the value of weather forecasting, it is virtually inevitable that we will have to develop some sort of empirical basis for estimating the value of the forecast product [i.e., descriptive studies; Murphy (1994)]. Note that the value contributed by weather forecasting is not one of generating income for users. Users certainly can improve their profits by making use of weather information, but they generate income from selling their product or service. The only people who generate income from forecasts, per se, are forecasters who charge users directly for their services. Instead, users of weather information benefit from forecasts by minimizing their costs and losses on the basis of the forecasts. Instead of being able to point to income generated, we usually can hope only to offer information about resources saved. Such information often is difficult to document. In effect, we have to know what the costs and losses would have been in the absence of the forecasts, or in comparison to some other source of weather information (e.g., climatology or persistence forecasts). It is very useful, as well, to know the sensitivity of the value to the accuracy. It may be that costs and losses are very sensitive to weather, but not very sensitive to the weather forecast. For example, a hailstorm can cause tremendous crop losses, but there is very little at present that can be done to save the crops from those losses, given the relatively short lead times (generally less than 1 h) for severe thunderstorm warnings. A farmer might not be able to protect the crops, but investing in crop hail insurance is a decision that must be made. That decision is not particularly sensitive to the accuracy of hailstorm forecasts, however. Deciding to buy crop hail insurance is more a matter of knowing the climatology of hailstorms than one of forecast accuracy.

Presumably, budget cutting might lead eventually to accuracy losses, and it is important to understand how and to what extent this affects the value. Although the relationship between accuracy and value is not simple, as we have noted, it is unlikely that forecast value is going to improve for most users when forecast accuracy (no matter how this is measured, within reasonable limits) declines. If we have difficulty measuring quantitatively the precise value of the forecasts, we are further challenged to determine the tendencies in the accuracy–value relationship.

In his report on the cost–benefit of the NWS MAR, Chapman (1992) derived an estimate of the value of forecasts to users by citing a 1980 survey⁵ that asked what the respondents were willing to pay for weather services. The “resultant per capita figure” (presumably, some measure of central tendency of the responses) was$24.15 per year. Through an adjustment “for changes in purchasing power and population” this base figure became $35.50 per year. The benefit of the MAR was assumed to accrue through the absence of a 5% per annum “erosion of confidence” attributed to the obsolescence of the current system if it remained unchanged.

Chapman’s study also mentioned a questionnaire-based survey of “major agricultural, industrial, and commercial organizations,” asking the 250 respondents to estimate their weather-related losses, as well as to decide what fraction of those weather-related losses are preventable. It is not clear from Chapman’s report how or when this survey was carried out, how the results were extrapolated to the entire nation from this sample of commercial interests, or how the potential annual savings were broken down further into those due to“operational improvements” and to “scientific advances.” We are not prepared to dispute Chapman’s figures, but it should be clear that his report is not sufficient to address our concerns.

To obtain hard quantitative information, then, forecasters are going to have to develop working relationships with forecast users. In the National Weather Service, efforts along these lines are already under way, including the creation of advisory groups of users. However, those efforts generally are associated with obtaining user input about forecast products. Although we certainly favor seeking user input regarding forecast products, public weather services also need to find out from their users how to measure the economic value of the forecast products.

Forecasts can be of relatively low accuracy and still have value for an intelligent user, as we will show later. Whereas it should be obvious to public policy-makers that perfect forecasts would be quite valuable for all users, such perfection is unattainable. The public perception of weather forecast accuracy typically is more negative than the objective reality of forecast verification statistics. Many weather-forecast-sensitive users have learned how to take at least some advantage of existing forecasts, even with something less than perfect accuracy (as we discuss later). Historically, forecasters generally have not paid much attention to how their forecasts are being used, preferring instead to focus their attention on the meteorological aspects of the forecasting task.

It is our belief that this historical lack of attention to the economic value of forecasting is hurting the weather forecasting services (public and private) and, ultimately, the users of forecasts as well. In the public sector, which is under considerable budget pressure, the resource base for forecasting is being threatened; this may or may not have a negative impact on forecast quality. If forecast quality is perceived by the policy-makers to be diminishing or not worth the investments, then further budget cuts are likely. Thus, a downward spiral could occur, possibly leading to an evisceration of public forecasting services (e.g., as in New Zealand).

In any case, both public and private sector forecasters can benefit from having an accurate knowledge of the value of their products, at least to a representative cross section of important users. We certainly are not alone in these concerns. Recently, Emanuel et al. (1995) have noted that “At present, we have very little understanding of the costs and benefits of weather data and forecasting information.” They recommended “the commission of a collateral economic analysis of the costs and benefits of weather forecasting improvements” and“the establishment of an independent scientific committee for providing guidance to the National Weather Service. . . .”

The American Meteorological Society has established a committee on Societal Impacts and the National Center for Atmospheric Research now has a working group to address the societal impacts of weather, in collaboration with a diverse group of sponsors, including the U.S. Weather Research Program. They recently completed a workshop and a summary of that meeting⁶ can be found in Pielke and Kimpel (1997). We are wholehearted supporters of these efforts. A key to their success will be multidisciplinary involvement with economists, social scientists, private sector business, etc.

Establishment of the accuracy–value relationship is complex and begins with a clear knowledge of forecast accuracy. Unfortunately, most forecasters at present have a less-than-adequate understanding of the accuracy of their forecasts, because most forecast verification has focused on measures-oriented schemes. Again, this is outside the domain of this essay, and we have written about this elsewhere (Brooks and Doswell 1996). We note that the issue of multifaceted accuracy measurement is touched upon by Murphy and Ehrendorfer (1987) in their discussion, and it adds considerably to the complexity of establishing the accuracy–value relationship.

3. Some approaches

A recent study by Robinson (1989) has shown that it is possible to isolate the economic impact of weather events on weather-sensitive users. However, Robinson’s work stops short of finding out what the actual user costs are; rather, those costs have been estimated. Robinson’s interesting study also does not assess what the impacts might have been had the forecasts not been available. There are good reasons for a meteorologist’s inability to determine all the direct and indirect impacts of weather-related costs and losses for even a representative sample of users. It is also easy to understand the difficulty in estimating the user response in the absence of the forecasts. Nevertheless, these issues must be explored if we are to obtain credible information about the value of forecast products.

If establishing forecast value with some accuracy is to be accomplished, how might this be done? We believe that simply polling users with a question like “How valuable are weather forecasts to your operation?” or“How much are you willing to pay for weather information?” is not adequate. Even if users have clear subjective opinions about the value of forecasts they are receiving, this type of information is not going to be enough to satisfy the budget cutters. What is needed is an objective, quantitative knowledge of what the negative impacts on users would be if the forecasts were not available or were reduced in accuracy. That is, we need to document the costs and losses incurred as a direct consequence of being without the forecast information and/or if the forecast accuracy declines.

At the moment, the forecasting community is only beginning to develop an infrastructure to acquire such data, so it is important to consider how to go about acquiring it. Moreover, not all business users are interested in sharing information about the quantitative value of forecasts to their operation, since they believe it gives them a competitive edge. Fortunately, some creative and innovative ideas are already in the literature to point the way, and we are going to suggest some other ideas, as well.

4. Discussion

Given that there is a high probability of large budget cuts being imposed on the NWS in the next 5–10 years ($37 million in fiscal year 1997), it is imperative to develop quantitative estimates of the value of forecasts and the sensitivity of that value to the quality of the forecasts. This information can be used in a variety of positive ways, not simply to argue for maintenance of the status quo. Obviously, forecaster–user collaboration can be beneficial to both sides.

The important issue is the development of effective, meaningful bases for making cost-cutting decisions. If imposition of budget cuts is inevitable, then it behooves public decision-makers to be selective about applying those cuts, rather than simply imposing uniform budget reductions across the board. If the value of the forecasts suffers as a result of quality changes imposed by resource reductions, concern for public welfare demands that the least valuable components of the forecasting system need to be the primary targets of the budget cutters. Naturally, in this area, casualty reduction might take precedence over pure economic value; we are not prepared to make this sort of decision, but it is typical of the problems faced by public decision-makers.

In anticipation of cost-cutting exercises to come, we want to emphasize the multidimensional nature of forecast quality. As of this writing, there is a serious exercise under way to determine the impact of removing half of the nation’s rawinsondes, as a way to save resources. If the measure of the impact is the 500-mb anomaly correlation score, the assessment might have a much different implication than if the measure is the equitable threat score for precipitation or the area critical success index for tornado watches. The impact of an observing system can be global, national, regional, and local, simultaneously. Who is to decide what is the most important? If the proposed changes affect the global or national forecasts very little but have a huge impact on the mesoscale or convective-storm-scale forecasts, how is the decision to be made? Murphy and Winkler (1987) and Brooks and Doswell (1996) have emphasized the pitfalls of single scalar measures for forecast quality evaluation. Exercises to measure impacts of cost-cutting proposals need to consider the full range of potential adverse effects.

Moreover, it has been argued elsewhere (see BFD96) that a key issue is forecaster performance on the relatively few days when the weather attains hazardous proportions. The impact on forecast quality of a hypothetical reduction in rawinsonde flights on a day like the so-called Superstorm of March 1993 might be much larger than on a set of more or less “average” weather days. Day-to-day performance is certainly a substantial issue, but it would be a serious mistake not to consider the impacts of any particular budget choice on a set of days when significant hazardous weather developed. History offers some examples. As a cost-saving measure, routine ship rawinsondes in the northern Pacific were terminated in 1981, in spite of the study by Spagnol et al. (1980) showing a negative impact. In their paper, Spagnol et al. (1980) indicated that some measures might be useful to compensate for the loss of the soundings. It seems now that those compensating procedures may not have been successful; Lord (1996) has shown recently that there can be considerable numerical weather prediction model forecast sensitivity to the initial conditions in just this northeastern Pacific Ocean area where the ship formerly took observations.

Cost-cutting measures can take a variety of forms, some of which include large reductions in staffing. As noted in BFD96, it is possible to envision revised structures in the public forecasting system that would have reduced staffing but with a considerably changed overall character to the public forecasting task and the skills needed by the staff. In view of the unknown nature of the effects of budget reductions even within the present system, it is difficult at present to anticipate what the effect on forecast accuracy of any proposed changes might be, including the NWS’s MAR. Any accuracy changes are likely to be important economically, but in the absence of documented evidence for expected changes in either accuracy or value, it is hard to imagine how our policy-makers can make intelligent decisions in the current and future budget situations. Impacts on the quality of the public forecasting service can have a large “ripple” effect since, ultimately, weather forecasting is a primary reason for supporting much of the basic and applied meteorological research in all sectors. This is a situation we consider to be important to the entire meteorological community and, ultimately, for the nation as a whole.

The significance of this issue seems to call to a wide range of interests to support the burgeoning efforts already under way. We will need a coalition to develop among meteorologists (forecasters and researchers, public and private), users, and representatives from related fields (economics, policy-makers, etc.). We believe that, although the entire meteorological community ought to be concerned with the outcome of that decision-making process, we cannot and should not try to do this in meteorological terms only. We meteorologists tend to be poorly versed in techniques outside our particular domain of expertise that are going to be needed to carry out this effort.

Public policy-makers must make difficult economic decisions that include issues of human safety, as well as purely economic factors. We hope those decisions will be made in the United States with as comprehensive a knowledge of the economic impacts of weather forecasts as possible, rather than without that quantitative information.