Winter storms making landfall in western North America can generate heavy precipitation and other significant weather, leading to floods, landslides, and other hazards that cause significant damage and loss of life. To help alleviate these negative impacts, the California Land-falling Jets (CALJET) and Pacific Land-falling Jets (PACJET) experiments took extra meteorological observations in the coastal region to investigate key research questions and aid operational West Coast 0–48-h weather forecasting. This article presents results from a study of how information provided by CALJET and PACJET was used by National Weather Service (NWS) forecasters and forecast users. The primary study methodology was analysis of qualitative data collected from observations of forecasters and from interviews with NWS personnel, CALJET–PACJET researchers, and forecast users. The article begins by documenting and discussing the many types of information that NWS forecasters combine to generate forecasts. Within this context, the article describes how forecasters used CALJET–PACJET observations to fill in key observational gaps. It then discusses researcher–forecaster interactions and examines how weather forecast information is used in emergency management decision making. The results elucidate the important role that forecasters play in integrating meteorological information and translating forecasts for users. More generally, the article illustrates how CALJET and PACJET benefited forecasts and society in real time, and it can inform future efforts to improve human-generated weather forecasts and future studies of the use and value of meteorological information.
Winter storms making landfall in western North America can produce heavy precipitation, strong winds, and other significant weather, with severity similar to that found in tropical storms (e.g., Ralph et al. 1999; McMurdie and Mass 2004; Neiman et al. 2004). The resulting flooding, landslides, and other hazards cause substantial damage, deaths, injuries, and economic disruption (e.g., NCDC 1993–2004; Ross and Lott 2003; NRC 2004). Improving forecasts of these storms and associated hazards may help alleviate these negative effects. However, accurately forecasting these events remains challenging, due in part to the complex coastal topography and the limited observations available over the Pacific Ocean (e.g., Ralph et al. 1999, 2003; McMurdie and Mass 2004).
To address these issues, the California Land-falling Jets (CALJET) and Pacific Land-falling Jets (PACJET-20011) experiments were conducted during the winters of 1997/98 and 2000/01, respectively. The programs collected additional meteorological observations along the U.S. west coast and over the eastern Pacific, both for research and to aid operational 0–48-h weather forecasting. CALJET and PACJET facilitated research on West Coast winter storms, precipitation, and flooding (e.g., Ralph 2003; Neiman et al. 2004; Ralph et al. 2005, and references therein) and provided data in real time to National Weather Service (NWS) forecasters. The programs also interacted with forecast users, both directly and through NWS forecasters. CALJET was strongly supported by several user groups in California (Koch and Ralph 2000), suggesting that the program benefited society, but evidence for this benefit was anecdotal. Thus, PACJET-2001 included a systematic, more in-depth study of the societal impacts of the programs.
The study focused primarily on how CALJET–PACJET observations influenced NWS forecasts of West Coast winter storms, rainfall, and flooding and related user decisions. To understand the context in which CALJET–PACJET information was used, the study also investigated how NWS forecasters and users use information in general. In addition, the study examined how researchers and forecasters interacted with each other and with users. The results were developed by analyzing qualitative data collected from more than 70 h spent observing NWS forecasters and from interviews with NWS personnel, CALJET–PACJET researchers, and users.
How forecasters use information has previously been examined through decision-making experiments (e.g., Allen 1982; Stewart et al. 1989; Lusk et al. 1990; Heideman et al. 1993), statistical analyses of forecasts (e.g., Clemen and Murphy 1986; Roebber and Bosart 1996), descriptions of the human forecasting process in case studies (e.g., Funk 1991; Schlatter 1985; Andra et al. 2002) and in general (e.g., Doswell 1986a, b; Browning 1989; Corfidi and Comba 1989; Targett 1993), and studies of naturalistic forecaster decision making and forecast system design (e.g., McArthur et al. 1987; Hoffman 1991; Pliske et al. 1997). The approach used in this study is related to the last two approaches in that it investigates how forecasters behave in their natural work environment. Our results complement and update those from previous studies by systematically documenting the many types of information (formal and informal) used by NWS forecasters in the early twenty-first century. While aspects of this topic have been addressed in earlier work, we discuss several features of the human forecasting process not previously documented for the nonforecasting community. Our results also extend previous work by formally examining how forecasters use added information and the forecaster–user interface.
Section 2 presents the data collection and analysis methodology. Section 3 examines how NWS forecasters used and combined different types of information when generating forecasts during PACJET-2001. These results elucidate some ways that human forecasters still improve upon computer-generated forecasts (e.g., Reynolds 2003), well into the numerical modeling era. They also illustrate some key human contributions to the “person–machine” forecasting mix that has been debated for decades (e.g., Snellman 1977; McPherson 1991; Brooks et al. 1992; Targett 1993; Mass 2003a).
Section 4 examines how NWS forecasters used CALJET–PACJET observations in potentially hazardous weather situations. Section 4 also discusses several challenges program researchers faced in interacting with forecasters and how these challenges were overcome. Section 5 examines how CALJET–PACJET-related forecast information is incorporated into emergency management decision making, using a case study to focus the discussion. Together, these results illustrate qualitatively how CALJET and PACJET benefited NWS forecasters, California emergency managers, and society. The results also illustrate the important role that forecasters play in communicating forecast information to users.
Note that the study was conducted before the NWS implemented the National Digital Forecast Database (NDFD) and Interactive Forecast Preparation System (IFPS; Glahn and Ruth 2003). Although new forecasting technologies are often introduced, the NDFD and IFPS have significantly altered how NWS forecasters generate everyday forecasts. Thus, details of the forecasting process described here may have changed. Nevertheless, based on recent discussions with NWS personnel, we believe that many of the results remain relevant. The results also provide a baseline for studying how NDFD and IFPS have affected forecasters.
The article closes by summarizing and discussing key results. These results can help meteorologists design future programs with goals similar to CALJET–PACJET, as well as future efforts to improve the skill of human-generated forecasts and increase their usefulness. The results can also inform future studies of the use and value of hydrometeorological observations and forecasts, particularly those associated with field experiments and other limited-duration programs.
The study methodology focused on the collection and analysis of qualitative (nonnumerical) data, as described below. The formal data collection and analysis were performed by the first author. The second author contributed to planning the study and presenting the results.
a. Data collection
Data were collected using two primary methods: participant observation and semistructured qualitative interviews. Supplemental data were gathered from documents (including NWS products such as Area Forecast Discussions), informal interviews and discussions, interactions with PACJET participants, and a site visit to the Pescadero, California, region. Most data were collected in person during February and early March 2001. The use of information during CALJET was therefore investigated retrospectively. The use of information during PACJET-2001 was investigated primarily in real time, with a retrospective component when people revisited earlier events.
Participant observation is the “process of learning through exposure to or involvement in the day-to-day or routine activities” of people in a setting (Schensul et al. 1999, p. 91; see also Jorgensen 1989; Yin 1994). In participant observation, as employed in this study, the researcher observed forecasters conducting their job duties in their natural work setting, interspersing questions to gain insight into forecasters’ actions and perspectives. Forecasters were also asked to describe in real time (when possible) what they were doing and why. As the researcher built rapport, many sessions evolved to include periods of informal discussion and interaction.
The 71 h of participant-observation sessions are summarized in Table 1. (Tables 1 and 2 do not provide specific dates, locations, or job titles to preserve the anonymity of study participants.) Although sessions focused primarily on one forecaster role, most included observations of and interactions with multiple forecasters in a variety of roles. The sessions were selected primarily by time and location, although an effort was made to interact with different forecasters (within the limited data collection period). Given the forecasting roles selected as primary foci, the sample is likely weighted toward forecasters with more experience and greater interest in new information and technology.
In semistructured interviews, the researcher asked open-ended questions, using an interview guide but allowing for flexibility, follow-up questions, and discussion (e.g., Weiss 1994; Schensul et al. 1999; Bryman 2001). Ten individuals were interviewed, as summarized in Table 2. Interviewees were selected based on their participation or interest in CALJET–PACJET or on recommendations from others [the snowball method; e.g., Weiss (1994)]. The interviewees were selected not as a representative sample, but rather to gather a range of perspectives.
Interviews generally lasted 1–2 h. Some included a tour of the interviewee’s facility, providing further data. Several interviews included general discussions of the current and potential use of information by forecasters and/or users. Informal interviews and discussions were conducted with additional PACJET participants, Hydrometeorological Prediction Center (HPC) and Weather Forecast Office (WFO) personnel, and users. Because of the limited data collected from forecast users, results based on these data should be considered exploratory.
Questions asked in participant observation sessions and interviews evolved during the study, as early data collection helped focus the research. Participant observation and interview data were recorded by taking written notes, filling in details as necessary afterward. Notes from informal interviews and discussions were written in real time when possible, or soon thereafter. To minimize researcher intrusiveness, no sessions were tape- or video-recorded. Although the data are not verbatim transcripts, the study involved repeated interactions with many people, permitting data checking and clarification when necessary.
b. Data analysis
The qualitative data obtained from the participant-observation sessions, interviews, and other sources were analyzed inductively: within the general topics of interest, the analyzing researcher allowed specific concepts and relationships to emerge from the data. To do so, the researcher iterated through a cycle of reading through the data, developing or modifying codes (themes) to describe and organize the data, coding (indexing) the data by theme, reorganizing the data by code, rereading data, and so on. Some data were also analyzed by iteratively developing diagrams such as Fig. 1, to further clarify concepts and relationships. This analysis cycle was revisited during the writing process, as ideas evolved. Through many such iterations of discovering or revising ideas and testing them against the data, the concepts and relationships presented in the article developed. [See, e.g., Miles and Huberman (1994), Lofland and Lofland (1995), and Coffey and Atkinson (1996).]
Several techniques were employed to increase the validity and reliability of the results (e.g., Miles and Huberman 1994; Yin 1994; Huberman and Miles 1998). First, data were recorded consistently and revisited multiple times during analysis. This helped reduce researcher bias by countering, for example, human tendencies to overweigh information that fits preconceived ideas or leaves a strong impression. Second, data on similar topics were collected from different sources, for different locations, times, and situations. This sampling strategy allowed triangulation, that is, comparison of data from different sources and situations to clarify, refine, and corroborate concepts and relationships. This sampling also enhances the generalizability of the results. Third, to check how well the results represent processes and perspectives, the authors obtained feedback from one of the interviewees, a key informant with experience with CALJET–PACJET, NWS forecast processes, West Coast prediction, and the use of forecasts.2
3. General use of information by operational NWS forecasters
This section discusses how the NWS forecasters studied used different types of information during PACJET-2001. The results are presented both to provide a context for the results in sections 4–5 and to update and formally document important aspects of the human forecasting process that are unfamiliar to many members of the meteorological community.
Details of the human forecasting process vary with the individual, meteorological situation, forecast product, and location. They also vary with time, as available information, procedures, and products evolve. To produce results relevant beyond the locations and times observed, analysis of the results focused on commonalities across forecasters and situations. Aspects of the results that apply to only a subset of forecasters are noted as such.
a. Use of information in the weather forecast production process
1) Meteorological and related observations
Meteorological and related observations are a major source of weather information for forecasters. Types of observations that forecasters used and discussed include the following:
observations from regularly available platforms, such as radars, satellites, radiosondes, surface stations, lightning detection, rain and stream gages, buoys, and ships;
special CALJET–PACJET observations, such as wind and melting-level observations from coastal wind profilers; surface observations at wind profiler locations; and flight-level, dropsonde, and radar observations from National Oceanic and Atmospheric Administration (NOAA) P-3 aircraft flights during intensive observing periods (IOPs); and
ad hoc observations made by forecasters (on and off duty) and local spotters.
The observations used in a specific forecast depended on what observations were available as well as the individual, forecast situation, forecast product, and lead time.
When generating very short-term (0–6 h) forecasts, forecasters used observations to identify current weather features that could be extrapolated or interpreted to predict future weather. When generating 12–24-h or longer forecasts, forecasters primarily used observations as indirect input, for example, to assess model accuracy and bias [section 3a(4)]. Observations also enter forecasts indirectly through other types of information in Fig. 1, including numerical model output, climatology, and forecaster knowledge and experience.
2) Numerical model output
Another major source of information for forecasters is numerical model output. The forecasters studied used the suite of NOAA/National Centers for Environmental Prediction (NCEP) models most regularly, but some also used or discussed European, Canadian, and U.S. Navy models, as well as model output from universities. Although there were general similarities, the specific ways that forecasters used model output varied widely with the individual. Many forecasters expressed preferences for or against certain models or fields, and most had a preferred mode of using model output but adjusted as needed for the situation.
Forecasters sometimes used model output fields as direct input to forecasts, for example, looking at model precipitation forecasts when forecasting precipitation. However, they also used model fields and diagnostics extensively to identify future meteorological features, such as low pressure systems, fronts, or areas of vertical motion, that might affect future weather. After diagnosing these future situations, forecasters used their knowledge and experience to infer what weather these situations might produce. In this sense, forecasters sometimes used the model output as a form of “future observations.” They also usually filtered the model output through their interpretations of model accuracy and bias.
The forecasters (at the time studied) focused primarily on nonensemble model output. However, they usually adopted a simple ensemble strategy by comparing output from different models and model runs.
3) Postprocessed model output
Given the study’s emphasis on forecasts of precipitation amounts and landfalling storms in complex terrain, the forecasters studied did not generally use postprocessed model output such as the NWS model output statistics. However, several forecasters and interviewees mentioned the importance of improving model postprocessing algorithms to help forecasters adjust for model deficiencies and biases. California–Nevada River Forecast Center (CNRFC) forecasters relied extensively on a postprocessing algorithm (developed by a CNRFC forecaster) that adjusted model-predicted precipitation for topographical effects in the western United States. As this illustrates, model postprocessing that addresses the specific needs of forecasters can be extremely valuable.
4) Assessment of model accuracy and bias
The forecasters interpreted computer-generated forecasts through the lens of their assessment of the models’ accuracy and biases. They generally used these assessments to choose a “model of the day” on which to base their forecast, and then to decide how to modify the model’s predictions of meteorological features and associated weather. Forecasters also used these assessments to evaluate forecast uncertainty.
To assess model accuracy and bias, forecasters integrated many types of information (Fig. 1). They considered how well the models were currently performing, by comparing output from different models, output from consecutive model runs, model forecasts with climatology, and model initializations or short-term forecasts with observations. They also considered how the models had performed recently, how the models typically performed in geographical regions and meteorological situations of interest, and how the models performed in general, based on their own knowledge and experience and on input from other forecasters. Some of these assessments were based on objective evidence, while others were subjective.
Several forecasters noted the importance of having experience with a numerical or postprocessing model to understand how it performs in different situations. They observed that frequent updates in NCEP models make it difficult for forecasters to gain this experience.
5) Meteorological knowledge and experience
Forecasters have access to large amounts of observational, model, and other data. To sift through, assimilate, and integrate these data, they used their meteorological knowledge and experience. They did so largely by recognizing familiar meteorological patterns, often using three- or four-dimensional conceptual or mental models of the atmosphere (e.g., McArthur et al. 1987; Browning 1989; Doswell 1992). Forecasters applied knowledge, experience, and pattern recognition to 1) infer meteorological fields from other fields (e.g., infer winds, divergence, and lifting from upper-level pressure contours); 2) select important meteorological features on which to focus their information-gathering, interpretation, and forecasting efforts [the forecast problem(s) of the day]; and 3) interpret what different information might mean for the weather forecast. Through the first two applications, forecasters limited the amount of data they needed to examine and focused their energy, increasing their efficiency.
Forecasters develop knowledge and pattern recognition skills from education, training (e.g., Spangler et al. 1994), relevant research results, and experience: experience watching weather evolve, combining information to generate forecasts, and seeing the accuracy of their own and others’ forecasts. Some of this knowledge and experience is codified, for example, in forecasting guidelines or “rules of thumb.” Other forecasting knowledge remains informal.
6) Geography and climatology
The forecasters used several types of geographical and climatological information, including knowledge of climatological timing of convection, coastline features and their effects on weather, and—especially important in the western United States—topography and its effects. They used this knowledge to assess model accuracy and bias, adjust forecasts for model deficiencies, add spatial and temporal detail to model forecasts, and add specificity to short-term forecasts. To account for topographical effects in western quantitative precipitation forecasts (QPFs), HPC and CNRFC forecasters also routinely used a gridded climatological precipitation dataset developed using the Precipitation-elevation Regressions on Independent Slopes Model (Daly et al. 1994).
7) Local knowledge
The forecasters, particularly those at WFOs, used their local meteorological knowledge. They also used knowledge of local geographical and societal factors that affect societal vulnerability to weather and use of forecasts. Examples include knowledge of locations prone to mud- or rockslides, locations of important roads, local water drainage characteristics, residents’ attitudes toward weather, and local decision makers’ needs. This local knowledge helps forecasters generate more specific local forecasts, decide when weather might threaten lives and property, and produce forecasts and warnings that are more useful to their communities. One forecaster explained this by noting that NWS WFOs exist because they have local links and can provide “local information for local users,” at any time of day or night.
8) User needs, desirable forecast attributes, and verification
Along with their perceptions of local needs (discussed above), forecasters also incorporated more general user needs into forecasts. One example discussed by several NWS personnel is issuing forecasts in time to be included in newspapers and TV newscasts. As one NWS manager explained, it is “better to get a good product out on time than a very good product late” because some late products can be useless. At WFOs, another example is issuing special weather statements by Friday afternoon when significant weather is expected during a weekend, to help the public plan travel and emergency managers plan staffing. At HPC, QPF forecasters noted that a positive bias is undesirable for RFC hydrologic forecasting. As a result, they were cautious about predicting large precipitation amounts when heavy convective precipitation was likely but its location was uncertain. This consideration of RFC needs likely decreased the value of HPC QPFs to users interested in large local precipitation amounts, but HPC QPF forecasters do focus on locally heavy precipitation in a complementary excessive rainfall product.
The forecasting process also incorporates general desirable forecast attributes, related to user needs. The data collected indicated five such attributes: timeliness, lack of bias, consistency among products, temporal continuity, and spatial continuity. NWS personnel occasionally mentioned considering these attributes. These attributes are also promoted by procedures built into the NWS forecasting process, such as a standard forecast production cycle, verification, forecaster interactions, use of recent forecasts, and interpolation routines.
Forecasters’ interpretations of user needs and desirable forecast attributes are often subjective. Nevertheless, incorporating these interpretations into forecasts likely enhances the value of forecasts to many users. Forecasters appeared to develop their interpretations of these considerations primarily from interactions with customers, discussions with other forecasters, and input from managers (e.g., the meteorologist-in-charge or branch chief).
Other than occasional conversations with users, forecasters primarily obtain feedback about the quality and value of their forecasts through forecast verification. Several forecasters mentioned verification, suggesting that it affected their forecasts. For example, several WFO personnel discussed the specific criteria used to verify warnings. At HPC, someone commented that “experienced [QPF] forecasters learn not to be heroes,” in other words, not to predict large precipitation amounts because they will likely locate them incorrectly, lowering their verification scores. This illustrates the importance of evaluating forecasts with well-designed verification measures (see also Davis et al. 2006). Such measures should represent users’ needs, reduce forecaster hedging, and generally promote forecast “goodness” (Murphy 1991, 1993; Wilson and Ebert 2005). Otherwise, forecasters (and researchers) may try to improve forecast quality in ways that do not enhance value.
9) Recent forecast
Information from the most recent previous forecast was incorporated into forecasts in two major ways. First, forecasters often started generating a forecast product by modifying an earlier product valid for the same time. Second, at each shift change, the outgoing forecaster discussed the forecast situation and his/her most recent forecast with the incoming forecaster. Building these procedures into the forecast production process increases the temporal continuity of forecasts and blends multiple forecasters’ interpretations in NWS products.
10) Other forecasters
The forecasters incorporated information from their forecaster colleagues in several ways. First, they transferred information at shift changes, through forecast products and discussion [section 3a(9)]. Second, forecasters in the same office during shifts discussed the forecast situation and their forecast products, through scheduled and spontaneous interactions. Third, forecasters in different NOAA/NWS offices interacted, by reading each other’s products and by discussing forecasts during scheduled regional coordination calls and impromptu telephone calls.
Through these interactions, forecasters incorporate others’ knowledge, experience, and interpretations into their forecasts. Nearly every NWS forecast product therefore combines multiple individuals’ interpretations, through a human-based ensemble. This complicates tracking how specific information is used in generating any given forecast. However, because group consensus forecasts are often better than an individual’s forecasts (Winkler et al. 1977; Clemen 1989; Baars and Mass 2005, and references therein), this human ensemble likely improves many forecasts. Interactions among forecasters also help forecasters learn from each other, and they enhance spatial and temporal continuity in forecasts and consistency among forecast products.
11) Integration and interpretation
To generate forecast products, the forecasters integrated and interpreted information, with the help of computer-based forecasting aids. This process has previously been described as a combination of the diagnosis of the weather situation, formation or application of conceptual models to understand the situation, and prediction of future weather (e.g., Doswell 1986a, b; Hoffman 1991; Pliske et al. 1997). An alternate description of the first two steps is hypothesis or scenario building and testing against data (McArthur et al. 1987; Hoffman 1991; Doswell 1992). These steps are iterative (Hoffman 1991; Doswell 1992) and may not be distinct (Doswell 1986b). Generally, this process involves examining multiple meteorological fields and the time-evolving situation.
As noted by previous observers of the forecasting process, for expert forecasters, integration and interpretation combine analysis and intuition, with visual pattern recognition and application of conceptual models playing key roles (Doswell 1986b, 2004; Targett 1993). In this process, forecasters combine qualitative with quantitative information, informal with formal information, imagery with other data (Fig. 1; Schlatter 1985; Doswell 1986b; Bader et al. 1988). Performing these tasks that are challenging for computers is a major contribution of forecasters to the forecasting process (e.g., Schlatter 1985; McArthur et al. 1987; Targett 1993).
12) Forecasting aids
Computer-based forecasting aids assisted the forecasters in three major ways. First, Advanced Weather Interactive Processing System (AWIPS) workstations and software (Seguin 2002) helped forecasters view, integrate, and interpret the large volume of observational and model data available (see also Hoffman 1991; Andra et al. 2002). Visual displays, for example, helped forecasters overlay information and view time loops (to recognize evolving patterns). Second, forecast production and voice generation software helped forecasters generate and disseminate products. Third, specialized detection algorithms notified WFO forecasters of potential mesocyclones, heavy rainfall, and rapid stream rises, helping them monitor rapidly evolving, potentially hazardous situations (see also Andra et al. 2002). Through these mechanisms, humans and computers work together to generate forecasts.
b. Weighting of different information and dependence on lead time
How forecasters weight different information—particularly observations and numerical model output—varied with forecast lead time (Fig. 2; see also Doswell 1986a; Browning 1989). For forecasts shorter than 3–6 h (the spinup time for most numerical models), forecasters relied on observations, using extrapolation, knowledge, and experience to predict how current weather was likely to evolve. For forecasts longer than 12–24 h, forecasters relied on model output, using observations primarily to assess model accuracy and bias. In between, the relative weight placed on observations and model output depended on the individual and situation.
Although the study focused on 0–48-h forecasts, several forecasters mentioned that for forecasts longer than 3–5 days, they had lower confidence and thus tended to weigh climatology more. Even when forecasters expressed low confidence in model output, however, they still generally based their forecast on some variation of a model solution. The dependence of modern forecasters on computer-generated information for all but very short-range forecasts was lamented by one of our interviewees as a “meteorological cancer” (see Snellman 1977). For 0–48-h forecasts, however, the forecasters we observed never based forecasts on model output alone, without interpreting it using observations, knowledge, experience, and other types of information in Fig. 1. Forecasters rely on models, but they also use intuition, local knowledge, understanding of user needs, and other information to add value to computer-generated forecasts.
4. Use of additional CALJET–PACJET observations by NWS forecasters
CALJET and PACJET provided observations in data gaps in landfalling West Coast winter storms. These observations were, however, only a small increment to the large volume of information already available to forecasters. The first part of this section examines how WFO forecasters used these additional observations, within the context provided by section 3. Although forecasters used multiple types of CALJET–PACJET observations in a variety of situations, this discussion focuses on how they used offshore observations from NOAA P-3 flights during IOPs, as potentially hazardous winter storms approached the coast. The second part of the section discusses how the programs overcame several challenges they faced in providing new observations to forecasters (across all situations).
a. Use of CALJET–PACJET observations at WFOs during IOPs
Satellite observations and numerical model output usually alert West Coast forecasters to approaching storms. The challenge is predicting how a storm will evolve and what weather it will produce where and when. Because numerical models can have significant errors over the Pacific (McMurdie and Mass 2004) and often poorly represent small-scale features, forecasters require offshore observations to provide “ground truth,” similar to that provided by local spotters over land. These observations help forecasters understand the meteorological situation, assess model accuracy and bias, and add details about local weather to forecasts.
Offshore observations are regularly available from several sources, all of which have limitations. Satellites provide good horizontal coverage offshore but limited information about vertical structure, particularly under clouds. With the resulting gaps in key regions of storms, these observations can easily be misinterpreted. Polar-orbiting satellites may also lack the desired temporal coverage. Coastal radars can observe storm structure, but only over land and in near-offshore areas. Radars also often scan over important near-surface features. Ships and buoys provide offshore surface observations, but only at a few locations. In between these sources of information, before a storm reaches the denser land-observing network, forecasters are left with questions such as: Where is the surface front, how strong is it, and how fast it is moving? What are the speed and direction of the low-level jet and surface winds? Is it raining at the surface, and if so, how much? What is the freezing level? Is there convection, and if so, what are the storms’ strength and structure? An example, with clouds obscuring the surface front and limited operational observations available nearby, is depicted in Fig. 3.
In the absence of CALJET–PACJET observations, forecasters assessed the meteorological situation and generated forecasts as accurately as possible by using their knowledge and experience to interpret the available information. When available, observations from CALJET–PACJET P-3 flights provided detailed information about an approaching storm in key offshore observing gaps (Fig. 3), answering the questions listed above. This helped forecasters adjust timing, location, and intensity aspects of their forecasts and make forecasts of potentially hazardous weather more specific. For example, Monterey, California, WFO forecasters issued an area forecast discussion on 28 February 2001 stating: “PACJET P-3 aircraft data instrumental in determining the strength of this front with radar imagery and pilot reports influencing the update of the public forecast. . . .” In several cases, CALJET–PACJET observations also helped forecasters issue watches and warnings with longer lead times. On 25 January 2001, Monterey WFO forecasters used P-3 radar observations to issue, in conjunction with the Storm Prediction Center, a rare severe thunderstorm watch that verified. Generally, the WFO can only issue warnings for severe thunderstorms once the weather reaches the coast. Another example is described in section 5. We did not systematically examine how CALJET–PACJET observations affected forecast skill. However, our impression was that in the majority (but not all) of the cases, forecasts improved.
Sometimes, despite the operational observing gaps, forecasters were able to assess offshore meteorological situations fairly accurately before receiving P-3 observations. The added observations then primarily corroborated their assessments, increasing their confidence. Forecasters’ confidence often influences how they communicate about forecast situations with users, in worded forecast products and conversations, which can influence users’ decisions (see below and section 5). Thus, even when CALJET–PACJET observations did not significantly alter the weather forecasted, they may still have affected forecasts, user decisions, and societal outcomes.
This chain of influence is illustrated by a situation observed in a WFO on a weekend evening during PACJET. Because a winter storm with the potential to produce heavy rainfall was approaching the coast, the WFO and local flood control agency had called in extra staff. As the storm moved closer, however, satellite and coastal radar observations suggested that it might be weakening. The lead forecaster looked through the available offshore observations for further information, asking several of the questions mentioned above but finding no clear answers. Based on his best guess, the forecaster decided to decrease the probability of precipitation and mentions of heavy rainfall in the next forecast update. Shortly thereafter, P-3 observations became available, confirming his decision to downgrade the forecast. After the forecast update was issued, the flood control agency telephoned and queried the forecaster about his confidence in the downgraded forecast, to help decide whether the threat was over and staff could be sent home. The PACJET observations likely affected how the forecaster discussed the forecast, which likely influenced the user’s decision.
As this discussion illustrates, qualitatively, CALJET–PACJET observations benefited NWS forecasters and forecast users. However, because the forecast generation, communication, and use processes are complex, understanding the value of the observations required in-depth study. This complexity makes it challenging to quantify the real-world influence of a few observations added during an ongoing situation.
b. Challenges to providing useful new information to forecasters and reasons for success
The study identified several challenges faced by CALJET–PACJET researchers in providing forecasters with useful new information in real time. First, once NWS forecasters have issued a forecast or warning, they generally prefer to update the product only if doing so will provide substantial new information to aid decision making. The value of information thus depends on when it is received. Researchers had to learn to plan P-3 flights to provide information at times useful to forecasters, both within the forecast cycle and relative to the storms’ approach.
Another challenge is that forecasters are often busy, especially in potentially hazardous weather situations, and they already have far more information available than they can use. Given this time pressure and data overload, forecasters tend to prefer information that is quickly accessible, familiar, readily understandable, and clearly relevant (see also Morss et al. 2005). As one NWS interviewee explained, to be convinced of the value of new technology, he and others needed to see a “clear connection to weather on the ground.” He added that large improvements are often required to convince people that a new product or idea is worth using. Forecasters and PACJET researchers sometimes had different views of what was relevant, understandable, and useful. To bridge this gap, PACJET researchers held workshops, visited WFOs, provided forecaster training, and interacted with forecasters during the program. Researchers also worked with forecasters to develop display formats and observational interpretations that focused on information important to forecasters, and they developed cases for forecasters illustrating how the observations could be used (L. Nance 2005, personal communication). Motivating forecasters to use CALJET–PACJET observations was particularly important because the data were not available in AWIPS. To access them, forecasters had to turn to a nearby computer with Internet access or communicate by telephone with the PACJET operations center.
Despite these challenges, CALJET and PACJET did successfully provide useful information to forecasters (section 4a). Four factors contributing to this success were 1) CALJET–PACJET researchers’ interactions with forecasters when planning and implementing the programs (described above), 2) researchers’ commitment to providing information useful for operational forecasting, 3) researchers’ adaptability, and 4) the researchers, forecasters, and NWS personnel who acted as “program champions” (Anderson-Berry et al. 2004). The first two factors were important because they meant that researchers listened to forecasters’ needs and incorporated them into the programs. This increased forecasters’ trust in the researchers and their interest in trying the new information. Researchers’ adaptability was important because it allowed them to test new ideas, learn, and adjust as the programs progressed. Program champions were important because they reminded forecasters about the new observations, encouraged forecasters to use them, and when necessary explained how to access and interpret them (see also Anderson-Berry et al. 2004). More generally, program champions gave the program a presence in NWS offices: PACJET observations were used more when and where champions were present and active. Without these four factors, the added observations would have been less used by, and less useful to, operational forecasters. Although we did not examine the long-term effects of CALJET–PACJET innovations, sustainability is an important topic to consider in such technology infusion programs.
5. Use of CALJET–PACJET-related forecast information by emergency managers
Next we examine the interface between weather forecasts and emergency management decisions. A case from CALJET, the Pescadero Creek, California, flood (2–3 February 1998), is used to focus the discussion and illustrate key points. Because the results focus on winter storms and California emergency management, we cannot generalize beyond this context. In addition, only two emergency managers (both at the county level) were interviewed for the study. The results should therefore be considered exploratory, for further investigation in future work.
Pescadero Creek flows from the coastal Santa Cruz Mountains south of San Francisco west through a largely rural area of San Mateo County, to the small town of Pescadero near the Pacific Ocean (Fig. 4). Because the Pescadero Creek basin can receive significant orographically enhanced rainfall, the creek floods to some extent in many years. The region is also prone to landslides when soils are saturated. The region has limited emergency and medical care services, and inland areas can only be accessed via the small Pescadero Creek Road. When flooding and/or landslides occur, the road is often impassable near the coast and upstream.
During the strong 1997/98 El Niño, California experienced significant precipitation. By late January 1998, soils were saturated in many areas. When a major winter storm was predicted for early February 1998, San Mateo County emergency managers identified Pescadero Creek as one of several areas of concern. On 2 February, several hours before the storm’s heaviest rainfall reached the coast, CALJET researchers on a P-3 flight communicated information about the storm’s low-level jet to the Monterey WFO. The P-3 observations confirmed NWS forecasters’ concerns about heavy rainfall and flooding, and they issued a flash flood warning with an unusually long lead time (Ralph et al. 2003). San Mateo emergency managers had sufficient confidence in the NWS forecast to begin assembling search and rescue crews and equipment and positioning them near the mouth of Pescadero Creek, in a location with access to the area at risk but out of harm’s way. Several hours later, heavy rain caused record flooding on Pescadero Creek, triggered landslides, and washed out area roads and bridges. The crews positioned earlier rescued 129 people using inflatable boats; only one area resident died in the incident. San Mateo emergency management personnel say that the additional forecast lead time provided by CALJET observations and the Monterey WFO saved multiple lives near Pescadero Creek because it allowed them to stage crews and equipment before the heavy rain disrupted transportation into the area.
Like weather forecasters, emergency managers use multiple types of information in decision making. Because emergency services often focus on responding to events, emergency managers rely heavily on assessments of the current situation, such as accumulated rainfall and current stream height. To predict when hazardous events might occur, emergency managers use not only weather forecasts, but also environmental cues (such as heavy precipitation) and informal benchmarks (such as flooding or landslides in specific regions). They also use their knowledge about the vulnerability of area populations and the effects of weather in their area. For example, the interviewees discussed local factors, such as logging silt, water releases from small reservoirs, and migrant farm workers living along creeks, that increased the risk that heavy rain posed for segments of their communities. They knew where problems were likely during heavy rain, sometimes down to the specific building. In the Pescadero Creek case, emergency managers combined weather forecasts, situational assessments, and local experiential knowledge to focus on Pescadero Creek and position rescue crews in an appropriate location before flooding began. The weather forecasts were critical, but so was the (subjective) experiential knowledge.
Emergency managers often progress through multiple decision-making stages as information about a hazardous weather event evolves. In the Pescadero Creek case, for example, emergency managers first identified the potential for a winter storm, then positioned crews, then activated those crews for search and rescue operations. A generalized version of this event decision cycle is depicted in Fig. 5. When no specific event requiring action is occurring or is on the immediate horizon, emergency managers prepare and plan. If forecasts, environmental cues, or other information suggest increased potential for a hazardous weather event, emergency managers increase readiness, for example, by placing crews on call and more closely monitoring the situation. If information builds, suggesting the event is likely, imminent, or occurring, emergency managers may initiate event-specific preparation, such as calling in personnel and positioning crews, and then activate emergency operations. Emergency managers must also decide when the threat has passed so they can demobilize emergency operations (which can be as important as deciding when to activate). Throughout the cycle, they may consider “what if” scenarios, to aid planning and reduce the likelihood of surprises.
Within each stage in Fig. 5, emergency managers gather information from multiple sources, often seeking corroborating evidence as they assess the situation and decide on possible actions. Depending on the decision-making stage, weather forecasts can play different roles, ranging from increasing awareness to delaying demobilization to providing information for scenarios. Although each decision may be a small increment toward a life-saving or otherwise societally beneficial action, each can have important effects on the outcome. In the Pescadero Creek case, for example, the decision to position crews several hours in advance was critical.
Meteorologists sometimes consider user decisions to be one-time yes–no decisions based on weather forecasts (as formulated, e.g., in the cost–loss decision model). For emergency management, an example is an evacuation–no evacuation decision in response to a weather warning. The above discussion, however, illustrates two ways in which emergency management decisions often do not fit this model. First, the “decision” may actually be a sequence of smaller decisions, or a buildup of information until a threshold for action is reached. Second, multiple types of information are considered, and different types of information may be most useful at different stages. Discussions with water resource managers and other forecast users, as well as results in Stewart et al. (1984), suggest that many other user decisions also do not fit the idealized one-time, yes–no decision-making paradigm.
Given the many factors they consider and the time pressure they often face, the emergency managers interviewed emphasized the importance of receiving clear, concise, easily understandable weather forecast information. In potential flood situations, they were interested primarily in rain amount and duration because these suggest how well flood control and stream systems can handle the water. They noted, however, that their forecast information needs depend on the situation and on area characteristics such as basin response time, transportation and flood control infrastructure, and population attributes. Emergency managers’ information needs also depend on their resources and responsibilities, which vary by jurisdiction (see also Morss et al. 2005). Forecasters can adjust for these different needs by adapting forecasts to the situation and by responding to specific information requests.
Emergency management decision making is also influenced by forecast confidence and trust in forecasters. In the Pescadero Creek case, for example, if NWS forecasters had been less confident in their flash flood warning or if emergency managers had trusted the forecasters less, emergency managers would probably have waited to position crews, with potentially deadly consequences. Forecasters’ confidence was due partly to the CALJET observations (see section 4a). Emergency managers’ trust was due partly to their experience with NWS forecasts and partly to their good working relationships with the WFO (as an institution) and WFO personnel (as individuals). When making critical decisions, the emergency managers interviewed often wanted to talk with a forecaster, to ask for his/her assessment and intuition. Data gathered from water resource managers and forecasters indicate that other forecast users also find personal relationships and interactions with forecasters to be important.
In the Pescadero Creek case, forecasters and emergency managers combined CALJET-related information with other information to make a sequence of decisions that collectively benefited society. Given the complexity of information use and decision making in this case, it would be difficult to quantitatively estimate the contribution of CALJET observations to societal outcomes. Moreover, because each extreme weather event occurs in a unique meteorological, geographical, and societal context, quantitative outcomes in one event cannot generally be transferred to other situations. Looking at decision processes along with outcomes, however, CALJET observations helped forecasters and emergency managers confirm their concerns about flooding on 2–3 February 1998, identify Pescadero Creek as a high-risk area, and position life-saving crews—all with sufficient lead time.
Forecaster–user interactions observed during the study (section 4a) and anecdotal evidence gathered by CALJET–PACJET participants suggest that CALJET–PACJET benefited user decisions in other situations, albeit more subtly. Quantitatively estimating this benefit, however, would require a much larger research effort. Because meteorological events with dramatic societal impacts (such as the Pescadero Creek flood) occur infrequently, a several-month program may not include even one such event. This suggests that a case study approach may often be useful for evaluating the societal benefit of limited-duration meteorological programs.
6. Summary and discussion
This article investigates the use of information by NWS forecasters and forecast users during the CALJET and PACJET-2001 field programs, based primarily on data gathered from observations of forecasters and interviews with NWS personnel, CALJET–PACJET researchers, and users. The results examine how NWS forecasters used information in general and how they used added CALJET–PACJET observations. The article also discusses interactions among researchers and forecasters and the forecasting–emergency management interface.
Over the last few decades, technological advances have led to debate over the most appropriate, most cost-effective roles for NWS forecasters (e.g., Snellman 1977; Allen 1981; McPherson 1991; Targett 1993; Roebber and Bosart 1996; Mass 2003a; Roebber et al. 2004). The article informs this discussion by elucidating the important roles that human forecasters play, particularly when weather places life and property at risk. These roles include
identifying and focusing on the most important forecast issue(s) of the day;
applying knowledge and experience to modify computer-generated forecasts and interpret their meaning for weather of interest;
integrating quantitative, qualitative, and informal information to generate forecasts;
adapting forecasts to local geographical, meteorological, and societal factors and to user needs;
communicating with users in ways that enhance the use of forecasts in decision making; and
providing a personal element to forecasts—a human with whom users can build trust.
Overall, forecasters act as intermediaries between the large volume of meteorological observations, model output, and knowledge available and users: they condense, interpret, and integrate data and knowledge from different sources to produce more user-accessible, useful forecast information, and they communicate that information to users. Through this complex, dynamic process, forecasters benefit forecast generation, communication, and use. The NWS forecast process also integrates the interpretations and skills of multiple forecasters, forming a human forecast ensemble.
The results discuss a range of information types incorporated into human-generated forecasts, those that are well known in the meteorological community and those that are less frequently discussed. In doing so, the article suggests ways to improve the person–machine forecasting mix, forecast quality, and forecast value. Each element in Fig. 1 plays an important role in many forecasts. To increase forecast quality, it is important not only to improve meteorological observations and numerical model output, but also to help forecasters manage and interpret data by addressing other aspects of the forecasting process. To increase forecast value, it is important not only to improve forecast quality, but also to improve forecasters’ understanding of user needs and to formally incorporate more value-related considerations into the forecasting process, for example, by developing and implementing more user-relevant verification techniques.
A limitation of the results in section 3a is that they are based on data from 2001, before the NDFD and IFPS were implemented. Although our results cannot address how the NDFD and IFPS have affected the human forecast process, a topic of current discussion in the NWS and the meteorological community (e.g., Glahn 2003; Mass 2003a, b; Colman et al. 2002), they do provide a baseline for systematically examining this issue. In addition, by analyzing the human forecasting process, the article can help the NWS and others understand the possible impacts of other proposed modifications to the forecast process.
The article discusses challenges faced by CALJET and PACJET and several strategies that helped the programs introduce new information into the forecast process. It also illustrates how the programs benefited society in real time by tracing how CALJET–PACJET information propagated through the forecast generation, communication, and use processes, focusing on cases of potentially hazardous winter storms. These benefits are examined largely through case studies because the programs had limited duration and, thus, included a limited sample of societally critical weather forecasts (see also Anderson-Berry et al. 2004). In addition, CALJET and PACJET introduced information into complex, multistage decisions that incorporated a lot of other information. Understanding the influence of this added information therefore required in-depth examination. The analysis indicates that the added observations helped forecasters increase forecast specificity and lead time and reduce forecast uncertainty. In some cases, these forecast modifications, when communicated to users, aided decision making.
Last, the article examines how forecasters interface with users and how emergency managers use weather forecast information. The results describe different stages of emergency management decision making and explore the complexity of how emergency managers integrate forecast information into their decisions. They also indicate the importance of forecaster–user interactions. While based on limited data, these results raise several ideas for further study. More in-depth investigation is needed, for example, of the role that users’ trust in forecasts and relationships with forecasters play in decision making. Another topic for future research is how forecast uncertainty and forecaster confidence interact with users’ decisions. Answering these questions can help meteorologists craft more useful forecast messages, particularly those communicating uncertainty.
Research methodologies similar to that presented here can be used to study a variety of aspects of forecasting and forecast use. Such studies, while relatively rare in the weather forecasting community, can generate rich, detailed knowledge about the processes examined. This knowledge is interesting in its own right, and it can help meteorological researchers and forecasters improve forecasts, provide more useful forecast information, and increase the benefits of weather forecasts to society. Such studies can also be used to analyze how well programs to improve forecasts are working and how they might be improved. They could therefore play a valuable role in the test and refinement loop in the proposed testbed concept (illustrated in Fig. 2 in Dabberdt et al. 2005).
In addition to those mentioned earlier, the study identifies several topics for future work. One is further developing systematic approaches to evaluating the societal costs and benefits of meteorological field programs. Such efforts will likely need to involve evaluation researchers in program planning, incorporate qualitative and quantitative methods, and collect perishable forecast and socioeconomic data in real time. To be comprehensive, such evaluations should also include the longer-term benefits expected from research advances (an area not addressed here). Another research topic is how well forecasters are able to evaluate model accuracy and make other subjective assessments that influence forecasts. Also of interest is a more detailed investigation of how forecasters build their informal, subjective knowledge of factors such as local weather–society interactions and user needs, and how this knowledge influences forecasts and forecasts value. Last, because the forecast process is continuously evolving, further study is needed of how forecasters adapt to new technology and incorporate new information into their forecast processes.
The authors thank the many NWS forecasters and personnel, forecast users, CALJET–PACJET participants, and others who participated in or facilitated the study. We acknowledge Kristen Koch and Roger Pielke Jr. for their help in initiating the study, Louisa Nance for her work in PACJET forecaster training and forecast operations, and David Reynolds for his valuable feedback on the results. Figures 1, 2 and 5 were prepared by Kristin Conrad, and Fig. 4 was prepared by Jennifer Boehnert. The NCAR Advanced Study Program provided REM with travel support for data collection.
* The National Center for Atmospheric Research is sponsored by the National Science Foundation
Corresponding author address: Dr. Rebecca E. Morss, NCAR/MMM/ISSE, P.O. Box 3000, Boulder, CO 80307. Email: email@example.com
The informant’s feedback did not alter any of the specific results, so it was not coded and formally analyzed.
PACJET continued in later years; this article focuses on the 2001 phase of PACJET.