Abstract

Many factors affect the extent to which forecasts inform emergency responses. In a survey based on the Theory of Planned Behavior (TPB), 207 U.S. emergency managers (EMs) were asked about 1) their past and intended future use of short-term weather forecasts and recorded weather data, 2) the perceived limitations and 3a) their attitude toward the usefulness of such weather information, 3b) their attitude toward their job and toward uncertainty, 4) perceived social norms, and 5) self-assessed numeracy. Work experience was found to be the best predictor of whether an emergency manager relied on recorded weather data and short-term weather forecasts in the past or intends to do so in the future. Among TPB variables, mainly social expectations and data attitude drive the reliance on recorded weather data and short-term forecasts. The EMs’ perception of the weather information’s limitations is related to their perceptions of what their social surroundings think. In sum, this article sheds light on when and why EMs use weather data and forecasts and how training can be improved.

1. Introduction

a. Use of hydrometeorological forecasts by emergency managers

Hydrometeorological forecasts are intended to provide their users with time to prepare for the events to come. In the case of emergency managers (EMs), weather forecasts can give the opportunity to evacuate people from endangered areas, prepare equipment to be able to respond quickly, protect or move valuable property and hazardous materials before a severe weather event hits, or advance coordination with interdependent responding agencies.

Given that each state has different regulations and is subject to different weather hazards, and emergencies are by definition unique, nonroutine events, it is difficult to study extreme weather emergencies and the decisions they led to. Unsurprisingly, the literature on the use of hydrometeorological forecasts in decision-making consists mainly of case studies and a few surveys.

A much discussed case study is Grand Forks, North Dakota, during the Red River flood in 1997. Despite a forecast of good quality, there was ~$1 to $2 billion in flood damage because the decision-makers did not realize how much uncertainty was associated with the National Weather Service (NWS) outlook (seasonal forecast) they relied on to prepare their city for the coming high-water levels. Even though emergency responders took into account some uncertainty, the people of Grand Forks could not imagine such a devastating flood. Instead, they anchored their expectations on the much more manageable: the predicted crest level, which was more in accordance with their past experiences (Pielke 1999; Morss and Wahl 2007).

Besides the Red River flood in 1997, Morss (2010) describes two more cases: a flash flood in Fort Collins, Colorado, in July 1997, and a flash flood in Pescadero Creek basin, California, in February 1998. In Fort Collins, heavy rainfall caused an unmonitored detention basin to overflow, derailing a train, flooding a trailer park, and leading to gas explosions and fires, killing several people. In the Pescadero Creek basin, experienced EMs wisely decided to “position emergency teams and dive crews near Pescadero before flooding and mudslides [caused by heavy rainfalls throughout the winter] blocked off transportation into the area” (Morss 2010, p. 89). Comparing these cases, Morss concluded experience greatly drives expectations and therefore preparations. If the event deviates from previous ones, people are often caught off guard (likely due to representative, availability, and anchoring heuristics; Tversky and Kahneman 1974). Forecasts seem to reinforce this process, by not providing what-if scenarios to make people think outside their experience. A successful forecast can help to manage expectations and to guide further information collection by smartly placing monitoring crews (Morss 2010).

There are a few empirical studies on forecast use in decision-making. In a survey of the general U.S. public, Morss et al. (2010) find most people associate weather forecasts with uncertainty and can sufficiently interpret probabilistic forecasts. However, people decide to take action at different probability thresholds. Empirically, Demuth et al. (2011) show the frequency with which the general public obtains weather forecasts depends on what the forecasts are used for, their attitude toward the forecast (i.e., importance of and confidence in the information), the accuracy of the forecast,1 their gender, their age, and how long they have lived in the area. In a laboratory study in which EMs from Oklahoma worked through a simulation of a severe storm, Baumgart et al. (2006) find EMs highly value many kinds of weather information during all stages of a storm. In the simulation, radar products and (storm) spotters’ observations were used frequently and evaluated as very valuable. In their descriptive framework, the authors illustrate EMs interact with the NWS at different stages of an emergency; this is a two-way interaction in many cases (Baumgart et al. 2006). Additionally, Baumgart et al. (2006) find more experienced EMs use weather forecast products more frequently.

In a previous interview-based study (Hoss and Fischbeck 2016), we found that use of hydrometeorological forecasts is driven by more factors than the desire to have the adequate information to make good decisions, that is, to make use of the lead time a forecast potentially provides. For actual decision-making, EMs seem to rely much more on radar and observational data, because they were acutely aware of the uncertainty associated with NWS forecasts. Last, EMs preferred using official weather forecasts rather than their own view on the situation to communicate with the public, because the forecasts represent the official assessment of the situation.

b. The Theory of Planned Behavior

When using forecasts to prepare for weather-related events, EMs consider a number of things such as their evaluation of the forecast, the social repercussions of using/not using various forecasting tools, etc. (Hoss and Fischbeck 2016). The Theory of Planned Behavior (TPB) captures all of these components.

Ajzen (2006), who originally proposed TPB, summarized its components as shown in Fig. 1. The dependent variable is a human behavior; in this case the use of forecasts and recorded weather data to prepare for weather-related events. According to TPB, three types of considerations combine to influence the agent’s behavioral intention. First, there are behavioral beliefs that inform the agent’s attitude toward a behavior, for example, whether it is beneficial for them. Second, the agent considers normative beliefs held by others, informing the agent’s subjective opinion on whether they should behave in a certain way. Third, the agent accounts for all factors that might help or hinder the considered behavior, determining whether the agent believes they have enough control to carry out the planned actions. Depending on the true extent to which the agent has control over their actions, these intentions result in the studied behavior (Ajzen 2006).

Fig. 1.

Framework of the Theory of Planned Behavior (Ajzen 2006).

Fig. 1.

Framework of the Theory of Planned Behavior (Ajzen 2006).

Besides wide use in public health and social sciences (Cohen et al. 2009; Moyano Dı́az 2002), TPB has been applied to weather forecasts. Together, Artikov et al. (2006) and Hu et al. (2006) investigated why U.S. farmers do not use climate forecasts more, despite their continuous improvement. Given the descriptive results of this study, Hu et al. (2006) report the perceived utility of the forecasts was low (attitude), and some social groups—especially crop consultants and spouses—are perceived to support the farmers’ use of forecasts (social norms). Additionally, farmers worried about the reliability of the forecast provider (control beliefs). Artikov et al. (2006) found the attitude indicating the perceived utility of the forecasts was the best predictor of actual forecast use. The authors conclude changing the farmers’ attitude is likely to increase forecast use more than addressing technical limitations of the forecast.

2. Method

a. Description of TPB survey among EMs

In the current study, a survey for EMs was devised to assess the four components of the TPB (behavioral intention, attitude, social norms, control beliefs) for two types of weather information: 1) observed weather conditions of the past few days and hours (called “recorded weather data”), and 2) short-term weather forecasts for the next few days and hours; similar to the survey among farmers by to Hu et al. (2006) and Artikov et al. (2006). Furthermore, questions to assess numeracy on the subjective numeracy scale as constructed by Fagerlin et al. (2007) were added to the survey. The complete questionnaire is available in the online supplemental material (section A).

To account for the diversity of natural hazards in a large country like the United States, six hazards were included in the study: flash flooding, river flooding, hurricanes, tornadoes, heat waves, and snow and ice storms. The participating EMs were asked to select the one hazard they found most difficult to respond to in the last 10 years, and they subsequently answered the questions about forecast use for that hazard. In this way, it is possible to examine whether the emergency managers’ environments have had any impact on the survey responses and the use of weather information.

The survey was distributed online in April and May 2014 to EMs throughout the United States. The survey was sent out through a variety of channels. The International Association of Emergency Managers (IAEM) published the survey call in their weekly newsletter that reaches ~10 000 emergency management professionals. Additionally, the survey call was posted to the discussion boards of the six LinkedIn groups for emergency management professionals. Furthermore, the survey call was sent to all e-mail addresses of EMs available on the websites of each state’s emergency management associations (EMA), the emergency management departments of each state’s administration and—if existing—the professional organization in each state. The number of e-mail addresses retrieved on those websites was ~2000, but it impossible to know how many EMs saw the survey call on platforms such as LinkedIn. Therefore, a response rate cannot be calculated.

Only U.S. EMs who were older than 18 could participate. As an incentive, a raffle in which four participants won a $50 voucher for their favorite store was added to the survey.

b. Hypotheses

The hypotheses are based on the components of the TPB. It is hypothesized that EMs rely more on the recorded weather data and short-term weather forecasts, if—compared to their colleagues—they do the following:

  • Evaluate the studied weather information as more helpful for decision-making (data attitude).

  • Perceive those products to have fewer limitations and assess themselves to have the ability to use them (perceived limitations, self-efficacy).

  • Put a greater emphasis on protecting rather than rescuing, and are more confident in coping with uncertainty (job attitude).

  • Experience more social pressure to rely on weather information (social norms).

  • Have a higher self-assessed numeracy.

3. Results: Emergency managers’ characteristics and their data use along TBP variables

a. Demographics

In total, 363 EMs started and 207 completed the survey. Most participants were county EMs (127) working full time (152), and had 5–10 years of work experience as an EM. Given that most were between 51 and 60 years old, it is likely this job is their second career. Indeed, 101 of the participants are professional firefighters (65), paramedics (23), or policemen (10), or they are in the military (3). Another 67 are volunteers in one of these functions (11, 8, 20, and 28, respectively; for distributions, see Fig. B.2 in the online supplemental material). EMs from all over the United States (Fig. B.1 in the online supplemental material) participated, with many from Florida (30), Kansas (24), and North Dakota (14), in particular; 12 states are not represented.2 Three-quarters of the participants (152) were male.

Of the 207 participating EMs, 48 have attended college but not graduated, 57 graduated from college, and 63 earned a master’s degree. To become an EM, the vast majority has taken courses, trainings, and workshops (186): 30 have college degrees related to emergency management, 58 have professional degrees, and five were self-taught. While our finding that 84% have had weather instructions at some point sounds reassuring, this has to be qualified by the fact that only 31% of courses EMs take pertain to weather-related events, while 78% of the emergencies faced by EMs are actually caused by weather (Weaver et al. 2014).

When compared to the demographic study of EMs by Weaver et al. (2014), our convenience sample is representative of the profession. Of Weaver et al.’s 1058 participants, 81% were male and 72% were older than 45 years. Fischer (1996) reports the typical local emergency management association director participating in his study was 50 years old and had 14 years of work experience in the field. As in our sample, a high percentage (78%) of Weaver’s respondents was college educated.

b. Past behavior/intentions

How much did you rely on NWS data to make any of the following decisions or carry out any of the following actions?

The dependent variable in this application of the Theory of Planned Behavior is “relying on recorded data and short-term forecasts in making different decisions.” As Fig. 2 shows (Table B.1 in the online supplemental material), recorded weather data and short-term forecasts are mainly used for monitoring the situation (determining event location, tracking event progress) and initiating a response [warning the public, activating the emergency operations center (EOC)], and less for deciding to implement measures such as initiating an evacuation or opening shelters. EMs used this kind of information least to decide when to deploy storm spotters, which is an activity designed to gather more data. EMs intend to rely more on recorded weather data and forecasts in the future.

Fig. 2.

Extent to which survey participants relied on the four types of weather data: past behavior and intended behavior regarding recorded weather data and short-term weather forecasts.

Fig. 2.

Extent to which survey participants relied on the four types of weather data: past behavior and intended behavior regarding recorded weather data and short-term weather forecasts.

Additionally, it was asked which type of data EMs would prefer, if they only could have one. Overwhelmingly, across all hazards, the participants opted for the forecast (Fig. 3). A possible explanation could be that they reckon they could get to know the weather of the past few days somehow, but would have a much harder time making projections for the future.

Fig. 3.

Percentage of EMs who prefer either recorded weather data or short-term weather forecasts, if they only could choose one.

Fig. 3.

Percentage of EMs who prefer either recorded weather data or short-term weather forecasts, if they only could choose one.

c. Perceived limitations of weather data (perceived behavioral control)

Please rate how much the following factors have limited your reliance on NWS data when responding to events.

In this survey, the component of the TPB called perceived behavior control is represented by the EMs’ perceived limitations of the recorded weather data and the short-term weather forecast. A distinction can be made between self-efficacy (i.e., one’s own ability to do something) and behavioral control (i.e., external factors hindering the agent). In their meta-analysis, Armitage and Conner (2001) found evidence that self-efficacy performs as well as perceived behavioral control in predicting intentions and behavior. Therefore, besides the extent to which characteristics such as accuracy of the recorded weather data and short-term weather forecasts limited their reliance on such products, we additionally asked the participants to assess to which extent their own ability to understand the weather information and to apply it to the situation at hand limited their use of weather information (self-efficacy).

For both recorded weather data and short-term weather forecasts, more than half of the respondents felt little to no limitations for most of the characteristics (Fig. 4; Table B.2 in the online supplemental material). Regarding recorded weather data (Fig. 4a), EMs felt their ability to understand the information and apply it to the emergency response as well as the unavailability and insufficiency or the irrelevance of the information for their area were the most hindering characteristics. When wanting to rely on short-term weather forecasts, the frequent changes of the forecasted event magnitude, the inaccuracy of the forecast magnitude, and the inaccurate forecasted timing were perceived to be most hindering (Fig. 4b).

Fig. 4.

Survey answers how much various issues limited the extent to which EMs relied on weather data in percent.

Fig. 4.

Survey answers how much various issues limited the extent to which EMs relied on weather data in percent.

d. Social norms

Please rate how much the following groups expect you to rely on NWS data when responding to an event.

Please rate how much you worry about criticism from the following groups when responding to an event.

The questions about social norms assess the extent to which the EMs feel pressure from their surroundings to rely on recorded weather data and short-term weather forecasts. In an earlier, interview-based study (Hoss and Fischbeck 2016), we found EMs consider the reactions of the public, the media and other types of accountability when responding to weather-related events.

The survey shows EMs think that locals such as EM colleagues, city employees, elected officials, and residents want them to rely on forecasts more than others (Table B.3 in the online supplemental material). Regarding the question to what extent they should rely on short-term forecasts and recorded weather data, EMs most often judge groups that lack local knowledge, that is, the training and workshop instructors, government officials, and the NWS, as not applicable (Table B.4 in the online supplemental material). At the same time, however, the opinions of professionals in the field, such as their emergency management colleagues, and employees of the NWS and of government agencies, are most valuable to EMs.

Multiplying the answer scores for those two questions (group expectations and group importance) results in the overall impact of each social group (Table B.5 in the online supplemental material). Training and workshop instructors seem to have the least influence on the EMs’ perception of whether they should rely on recorded weather data and forecasts. The opinions of their colleagues at the emergency department and in the town hall in general are most relevant to the EMs. This is followed by NWS employees, the media, and family and friends. Therefore, managing the use of weather data is most effectively done through peer education and with direct contacts between EMs and the NWS. The survey indicates the opinions of professionals with local knowledge are more valuable than the views of workshop and training instructors. Additionally, Morss et al. (2005) mention that practitioners prefer to talk to people with whom they have had long-term relationships when seeking scientific information.

Most EMs (60.7%) agreed taking into account the expectations of others is useful (Fig. 5a). Additionally, it is important to note that as much as 37.6% worry about liability when using weather information to respond to emergencies (Fig. 5b). This vote is less clear, featuring a bimodal distribution.

Fig. 5.

Survey answers to what extent EMs agree or disagree with two statements about valuing the opinion of others and worrying about liability (N = 162)

Fig. 5.

Survey answers to what extent EMs agree or disagree with two statements about valuing the opinion of others and worrying about liability (N = 162)

e. Attitude: Weather data

Across all questions, more than 80% agreed or strongly agreed using NWS data was beneficial in various ways. The one exception is the question about whether these data were the most important source of information (Fig. 6; Table B.6 in the online supplemental material).

Fig. 6.

Answers to data attitude questions (N = 161, 156).

Fig. 6.

Answers to data attitude questions (N = 161, 156).

f. Attitude: Job

The rationale for questioning participants about their job attitude was that many EMs are or have been firefighters or paramedics at some point. In those jobs, they were primarily trained to respond to, rather than to prepare for, crises. It is thus conceivable that they would make less use of forecasts because they are not as familiar with the opportunities that the lead time provided by forecasts presents.

A majority of EMs (65.7%) view their job to protect rather than to rescue citizens (Fig. 7a; Table B.7 in the online supplemental material). They are divided on the question of whether they are supposed to prevent or to respond to hazardous situations (Fig. 7a) and in their attitude toward uncertainty (Fig. 7b). Between 57% and 72% (strongly) disagree uncertainty hinders them in their work.

Fig. 7.

Answers to job attitude questions in percent. For the questions with the minus in front, the left side corresponds with a preventive mindset. For the other questions, the scale is reversed (N = 207). For corresponding tables, please see the online supplemental material.

Fig. 7.

Answers to job attitude questions in percent. For the questions with the minus in front, the left side corresponds with a preventive mindset. For the other questions, the scale is reversed (N = 207). For corresponding tables, please see the online supplemental material.

Additionally, we posed the binary question of whether EMs react rather intuitively or whether they deliberately consider the information and options that they have before taking action. Of the 207 participants, 29% assessed themselves as responding intuitively.

Using a hypothetic tornado case study, Weaver et al. (2014) found many EMs decide to react relatively late. For example, 22.2% would only come into action by the time the tornado had already caused damage, while 30.3% would initiate a full response after the Doppler radar has confirmed a tornado 30 min away (Weaver et al. 2014). Combining these results with ours, it can be concluded that EMs do distinguish themselves from professions like firefighters and paramedics in the sense that they would rather protect than rescue people, that is, EMs try to get people out of harm’s way beforehand. However, they do not see themselves as having to prevent hazardous situations. That might explain the late responses found by Weaver et al. (2014). Maybe, EMs simply regard 15 minutes as enough time to reach a tornado shelter. Any damage prevention beyond that might fall outside the perceived scope of the EMs’ task. Given that EMs do not seem to experience uncertainty as seriously hindering their work, the late response might also be meant to reduce the number of false alarms. In a previous study (Hoss and Fischbeck 2016), we found EMs do choose the weather information with the least uncertainty in them. But apparently, a majority does not think uncertainty prevents them from acting. Morss et al. (2005, p. 1596) explain “practitioners often deal with uncertainty by finding the best information they can quickly and easily obtain and interpret, making the decision required for the moment, and moving on.” [Emphasis added.] Hence, EMs might simply experience uncertainty not as a problem but as a daily given.

g. Subjective numeracy

In surveys, numeracy is usually assessed by asking people mathematical questions. When they encounter such questions, people often feel judged and tend to quit the survey. Additionally, it cannot be assured that participants do not use calculators or seek help from others when conducting a survey online or over the telephone. To address this problem, Fagerlin et al. (2007) developed and Zikmund-Fisher et al. (2007) validated the subjective numeracy scale (SNS). In an empirical study, they selected 8 of originally 49 questions that best correlate with established objective numeracy scales. Four of those eight questions assess the ability to conduct mathematical operations, for example, calculating a 15% tip. The other four questions ask participants whether they prefer numbers or percentages over words, etc.

Our survey participants were confident they can work with fractions and percentages and clearly preferred numbers (Fig. 8; Table B8 in the online supplemental material). EMs feel more comfortable working with percentages than with fractions. While they generally seem to prefer numbers over words, they do so slightly less when talking to people directly.

Fig. 8.

Survey answers numeracy including the correlation to objective numeracy and comparison to results reported in Fagerlin et al. (2007) (N = 207).

Fig. 8.

Survey answers numeracy including the correlation to objective numeracy and comparison to results reported in Fagerlin et al. (2007) (N = 207).

When comparing these scores with previous studies, EMs score considerably higher on self-assessed numeracy than participants in previous studies [Fagerlin et al. 2007 (comparison in Fig. 8); Paolacci et al. 2010; Zikmund-Fisher et al. 2007]. The large percentage of male EMs (73.4%) is the most prominent difference between our participants and those in the literature.

4. Results: The effect of emergency managers’ characteristics on their data use according to the TBP

a. Principal component analysis

To reduce the number of variables, principal component analysis (PCA) was run for each of the variables of the TPB, that is, for both recorded weather data and short-term forecasts: past behavior, future intentions, perceived limitations, social norms, job attitude, attitude toward weather data, and subjective numeracy. PCA is a statistical method to identify strong patterns in a dataset. It aggregates variables to arrive at fewer variables that are statistically independent. The used method (command “princomp” of R-package “stats”) computes the covariance matrix and determines the eigenvalues. Based on the scree plot, the eigenvalues, and nontrivial factor loadings, the best number of components to be included in the regressions was determined. Prior to the PCA, the variables were statistically standardized.

Taking the TPB variable “socials norms” as an example, the EMs rated the expectation of various social groups to which extent they should rely on weather information. The PCA returns loadings for each of these answers (see tables in section C of the online supplemental material), that is, regarding each social group. Each EM’s response to the survey question regarding each social group is multiplied with the corresponding loading (same loading for all EMs, different loadings for each social group). These products are then summed up for each EM to an aggregate score. In some cases (see below), two principal components were identified, for example, for social norms there is a general one and one distinguishing between officials and the public. In this case, the procedure that was just described was carried out twice, that is, for each of the two components. This results in two aggregated scores for each EM that are used in the further correlation and regression analyses.

Social norms, perceived limitations and job attitude have more than one component. The second component of social norms “officials versus public” separates EM colleagues, NWS and government employees, and training instructors from the public, such as residents, elected officials, and the media. For perceived limitations, the second component “data quality versus usability” contrasts inaccurate estimates of and frequent changes to event magnitude and timing with the unavailability of relevant information and limited user ability (self-efficacy). The third component of perceived limitations (“information availability”) sets limited availability of relevant data (and in the case of recorded data, also inaccurate estimates of event magnitude and timing) apart from limited self-efficacy. The second component of the job attitude “mindset versus uncertainty” contrasts the questions on a preventive versus a responsive mindset with those on feeling confident in uncertain situations. The second component of numeracy, “proficiency with numbers,” sets the preference for numbers or words apart from the confidence in working with fractions and percentages.

b. Correlations among TPB variables

This is section merely describes the statistically significant correlations among TPB variables. Together with the results of the regression analysis, they will be placed into a broader narrative in section 4d (summary).

1) Recorded weather data

The statistically significant correlations (Table D.1 in the online supplemental material) indicate EMs’ own perceptions are associated with their social surroundings. As the following three correlations suggest, they might (unconsciously) either assume that others think similarly to themselves or adopt the opinions of others: 1) A positive data attitude (general data attitude) is associated with more social pressure to rely on recorded weather data (social norms general) − r(151) = 0.31, p < 0.01; 2) EMs who generally perceive more limitations (general perceived limitations) experience less pressure from the public (public vs officials) − r(136) = 0.23, p < 0.01; 3) perceived social pressure to rely on recorded weather data (general social norms) is correlated with a higher self-reported ability to understand and apply the data, and better timeliness, availability for the area, regular releases, and sufficient information (usability vs quality) − r(136) = 0.14, p < 0.1, respectively.

The utility and perceived limitations of recorded weather data are mainly correlated with skills. EMs with a higher subjective numeracy (numeracy general) perceived fewer limitations (perceived limitations general) − r(140) = 0.21, p < 0.01. Additionally, those feeling more confident with fractions and percentages (confidence vs preference) tend to perceive the relevant data to be more available and accurate (ability vs product). In contrast, those with a less strong preference for numbers over text (confidence vs preference) have a greater feeling that their ability limited their use of recorded data (ability vs product) − r(140) = 0.15, p < 0.1. Unsurprisingly, a better perceived ability to use these data and a better perceived quality of the of the data (usability vs quality) coincides with a positive data attitude (general data attitude) − r(139) = 0.22, p < 0.01.

Finally, the negative correlation between a more preventive mindset (general job attitude) and data attitude indicates EMs who would like to plan ahead of time find recorded data unhelpful − r(158) = −0.16, p < 0.05. For EMs with a preventive mindset, availability and accuracy are more important obstacles than their own ability, as suggested by the negative correlation between general job attitude and information availability − r(140) = −0.14, p < 0.1.

2) Short-term forecasts

For forecasts, the statistically significant correlations provide additional indications that the EMs’ attitude is associated with their social surroundings (Table D.2 in the online supplemental material): 1) As before, a generally positive data attitude (general data attitude) correlates with more social pressure (general social norms) − r(144) = 0.17, p < 0.05. 2) For forecasts, a preventive job attitude (general job attitude) is associated with a less social pressure (general social norms) − r(149) = 0.14, p < 0.1, instead of a less favorable data attitude (general data attitude) as it is in the case of recorded data. 3) Similarly, being confident in uncertain situations (uncertainty vs mindset) is associated with feeling less public pressure to rely on forecasts (general social norms) − r(149) = 0.13, p < 0.1. 4) As before, EMs perceiving more general limitations (general perceived limitations) tend to feel less pressure from the public (public vs officials) − r(128) = 0.16, p < 0.1.

The EMs’ attitude toward short-term forecasts is mainly driven by perceived limitations, but those are interestingly not correlated with numeracy: Perceiving short-term forecasts to have more limitations (general perceived limitations) correlates negatively with a positive data attitude (general data attitude) − r(127) = −0.25, p < 0.01. The negative attitude toward forecasts is associated with perceived limitations of the accuracy and frequency of estimates of event magnitude and timing. But while the perceived limitations for recorded weather data relate to numeracy, in the case of short-term forecasts the subjective numeracy is not significantly correlated with the perceived limitations.

c. Regression analysis

In total, four regression models were built: one for past and intended future behavior for each recorded weather data and short-term weather forecasts. To reduce the numbers of variables, in a previous step, we tested one by one which demographic variables were statistically significant in the model consisting of the TPB variables. These turned out to be work experience, the type of hazard and EM education “other.” For the short-term weather forecasts, these additionally included having received weather instructions and a police background. All other demographic variables were excluded from the four main models. Multicollinearity was not a problem because the variance inflation factors for all independent variables in the regression models are between 1.0 and 1.5.

1) Recorded weather data

Work experience and winter storms are, by far, the dominant variables for recorded weather data (see Table 1). EMs appear to rely less on recorded data for winter storms than for other hazards (Table 1). Additionally, consistent with findings of Baumgart et al. (2006), EMs with less than one year of work experience rely much less on recorded weather information than each of the groups with more work experience. This supports Pielke’s and Conant’s (2003, p. 1352) opinion that “weather forecasts have value not because they are by any means perfect, but because the vast experience of users of those predictions fosters the incorporation of them into the decision routines.”

Table 1.

Regression results for recorded weather data with and without demographic variables. (The regression coefficient is denoted by Coef and standard error is SE.) The statistically significant correlations are printed in bold.

Regression results for recorded weather data with and without demographic variables. (The regression coefficient is denoted by Coef and standard error is SE.) The statistically significant correlations are printed in bold.
Regression results for recorded weather data with and without demographic variables. (The regression coefficient is denoted by Coef and standard error is SE.) The statistically significant correlations are printed in bold.

When demographic variables are excluded from the regressions, social norms (social norm general) and data attitude are the most important explanatory variables for relying on recorded weather data. These two variables are precisely the ones Artikov et al. (2006) found to be significant predictors for the use of climate forecasts by farmers. A possible interpretation is that work experience translates to the EMs having learned to listen to their social surroundings and having experienced the benefits of using such data to inform their decisions. Artikov et al. (2006) think farmers might decide whether to use climate forecasts after consulting others or might simply do what everyone else around them does. This could also be the case for EMs. For example, Morss and Ralph (2007) found EMs find personal relationships with forecasters very important when faced with a critical situation.

The survey results support three of our five hypotheses: EMs experiencing more social pressure, having a more favorable opinion of the data and a more preventive job attitude tend to rely more on recorded weather data. Contrary to the hypothesis, perceiving more limitations of the data (small effect) is associated with relying more on recorded weather data, too. Numeracy was found to be statistically insignificant.

2) Short-term forecasts

For short-term forecasts, work experience is the dominant predictor as well. Work experience has a stronger influence on past behavior than on intention for the future (Table 2). EMs with work experience of 5–10 years rely on forecasts the most. But interestingly, those with more than 10 years of work experience rely less on forecasts than those who have been EMs for less than 10 years. A possible explanation might be that, in contradiction to Baumgart et al. (2006) and Pielke and Conant (2003), EMs learn over time that strongly relying on forecasts is not beneficial. That does not necessarily mean they use forecasts less, but instead they might be looking for more information to supplement the forecasts.

For forecasts, TPB variables are statistically significant even when demographic variables are included in the regression (Table 2). Perceived higher social expectations (social norm general) resulted in higher reliance on the forecast. Additionally, EMs who perceived unavailability for the area, insufficient information, irregular lead times, receiving the forecast too late, and inaccurate estimates of event magnitude and timing (usability vs dissemin.) as more of a problem relied on forecasts less. This confirms the finding that perceived limitations of the forecast—rather than self-efficacy—influenced the use of recorded weather data and forecasts in the past but does not inform intentions for the future.

Table 2.

Regression results for short-term forecasts with and without demographic variables. The statistically significant correlations are printed in bold.

Regression results for short-term forecasts with and without demographic variables. The statistically significant correlations are printed in bold.
Regression results for short-term forecasts with and without demographic variables. The statistically significant correlations are printed in bold.

Unlike for past behavior, the overall subjective numeracy (numeracy general) is statistically significant for future behavior when demographic variables are included in the regression. Interestingly, feeling more confident regarding numeracy is associated with intending to rely less on the forecasts in the future.

The survey results support two of our five hypotheses: EMs experiencing more social pressure, and having a better opinion of the forecast rely more on it. Finally, while less reliance on recorded weather data was driven by a perceived unavailability of the relevant data, the utility of short-term forecasts was mainly limited by a perceived lack of accuracy and reliability of the data.

Contrary to the hypotheses, EMs with higher self-assessed numeracy tend to rely less on forecasts (small effect). Job attitude and the general perceived limitations of short-term weather forecasts are statistically insignificant predictors of the extent to which EMs rely on those forecasts.

d. Summary

The correlation and regression results can be aggregated to a more general picture (Fig. 9). Besides the dominant predictor work experience, social expectations and data attitude are significant predictors of the reliance on recorded weather data and short-term forecasts.

Fig. 9.

Correlations in the Theory of Planned Behavior model when including demographic data (solid black arrow) and excluding it (other arrow).

Fig. 9.

Correlations in the Theory of Planned Behavior model when including demographic data (solid black arrow) and excluding it (other arrow).

Social expectations and data attitude are correlated with the perceived data limitations. It is plausible that EMs reflect their own perception of the weather information onto their social surroundings when answering the survey questions. But it is also possible a positive attitude of the social surroundings influence the EMs’ attitude toward the data and their perception of its limitations.

At the end of the causal chain, higher subjective numeracy is correlated with fewer perceived limitations of recorded weather data. Possibly, EMs with higher self-assessed numeracy feel more confident to cope with the imperfection of weather information. However, this does not mean those EMs also decide to rely on such information more.

5. Limitations of method

Armitage and Conner (2001) found in their analysis of 185 studies that TPB accounted on average for only 21% of the variance in self-reported behavior and 39% in intention. Applying TPB to reliance on recorded weather data and forecasts yielded 16%–26%, and 35%–54% when demographic variables are included. There are several reasons why our TPB model does not explain more of the variance:

  1. Most importantly, the variables have been constructed by applying PCA to the original survey questions. PCA was necessary to reduce the number of variables, given the limited amount of data points. However, the choice of components cumulatively did not add up to much more than 50% of the variance for all TPB variables, except for perceived limitations. Consequently, a lot of the information inherent to the survey data would only be accessible with a much larger sample size.

  2. Conventional linear regression might not be the best mathematical method. Generally, some researchers question whether survey answers on a Likert scale can be treated as interval data in order to subject them to principal component analyses and linear regressions. Given that those survey answers are really ordinal data, correspondence analysis and profit regression should be used instead. In those analyses, each answer option is treated as a binary variable. Thus, a question with an answer scale from one to five becomes four variables instead of one. However, such a large number of variables cannot be accommodated by a dataset consisting of 207 participants. Therefore, in this case, the only choice is to revert to principal component analysis and conventional linear regression.

  3. The limited amount of variance the regression models are able to explain could be due to the highly variable nature of emergencies. In a previous study, for example, we found that often cascading events, such as collapsing infrastructure during a flood, rather than forecast uncertainty, are the most trying challenges for emergency responders (Hoss and Fischbeck 2016).

Additionally, it needs to be noted emergency managers with a stronger affinity for science might have answered the survey call. Especially, the high scoring in the self-assessed numeracy might have been impacted by that.

Finally, asking the emergency managers to answer the survey with one particular event of one of six hazard types in mind might have influenced the results. After all, the emergency responses to the different types are functionally different, that is, some hazards allow for a longer preparation time than others so, for example, evacuation exercises might be organized differently. The type of hazard was accounted for in the regression analysis.

6. Conclusions

Work experience was found to be the best predictor of whether an emergency manager relied on recorded weather data and short-term weather forecasts in the past or intends to do so in the future. Additionally, those products were relied on less for ice and snow storms than for other weather-related hazards. If work experience, that is, “school of life,” really is the main factor driving the reliance on recorded weather data and forecasts, emergency management training needs to be improved significantly, especially since 84% of the participants had received instructions on how to use these products at some point.

When demographic variables are excluded, as is done in most TPB studies (Cohen et al. 2009), social norms and data attitude become the two dominant predictors for recorded weather data and short-term weather forecasts for both past behavior and future intentions. It is possible that work experience results in learning to appreciate these types of weather information for decision-making and to listen to the social surroundings. Social norms and data attitude are correlated with the perceived limitations of the forecast, which—in the case of recorded weather data—in turn are correlated to subjective numeracy. However, it cannot be determined which direction the causalities take.

Going into more detail, it was found EMs rely little on recorded weather data and forecasts to determine when and where to deploy storm spotters and on recorded weather data when to initiate evacuations. Storm spotters are a way to gather more data early on. But apparently, worrying forecasts do not inform this decision much.

To rely on recorded weather data, the inability to apply information to the emergency response, the unavailability of the data for the area, and receiving the data too late are perceived to be most limiting. The former could be addressed by emphasizing the use of recorded weather data more in instruction sessions. For forecasts, the inaccuracy of and frequent changes to event timing and magnitude are perceived to be the greatest obstacles. This confirms that uncertainty limits the use of forecasts in practice.

Social pressure to rely on the studied weather information comes mainly from professional peers who possess local knowledge: EM colleagues, other city employees, and NWS employees. Training and workshop instructors are least influential. Therefore, localized training and peer education would be the most promising way to increase the value of weather information for decision-making. The EMs would appreciate it, because they have an overwhelmingly positive attitude toward those products.

One-third of the EMs thought they tend to act intuitively rather than considering all facts and figures extensively. A majority of 65.7% prioritized protecting citizens rather than rescuing them. This distinguishes EMs from professions such as firefighters and paramedics. But slightly more than half thought it was not their task to prevent hazardous situations. This is not surprising since three-quarters of the participants have a background in another emergency response profession. Consequently, there could be much room for improvement to make the most use of the lead time forecasts provide, especially since uncertainty was not considered much of a problem for planning an emergency response by about two-thirds of the participants (57%–72%).

When compared to the participants in previous studies that included the subjective numeracy scale, the EMs assessed their numeracy as much higher. Generally, they feel more confident in working with percentages than with fractions. Additionally, they tend to prefer numbers over text, unless it is in a conversation with someone.

Reflecting on the Theory of Planned Behavior itself, a distinction between perceived behavioral control (i.e., external factors limiting a behavior) and self-efficacy (i.e., internal factors limiting a behavior) was found to make a difference statistically, indicating that for this TPB component, external and internal factors should be assessed separately.

In sum, the paper sheds light on when and why EMs use weather data and forecasts and how training can be improved. Future research should verify whether training by professional peers can accelerate the learning curve in using weather information to prepare for emergency, that is, whether work experience can be transferred to some extent.

Acknowledgments

Most importantly, we thank all the emergency managers who took the time to complete the survey. Special thanks to those who sent additional comments. We also want to express our gratitude to the people who helped to distribute the survey, most notably the International Association of Emergency Managers, which published the survey call in its newsletter. Last, many thanks to Alex Davis for patiently providing advice on the appropriate statistical methods.

As to funding, Frauke Hoss was supported by an ERP fellowship of the German National Academic Foundation and by the Center of Climate and Energy Decision Making (SES-0949710) through a cooperative agreement between the National Science Foundation and Carnegie Mellon University (CMU). This article is a chapter from the corresponding author’s dissertation (Hoss 2014).

REFERENCES

REFERENCES
Ajzen
,
I.
,
2006
: Constructing a theory of planned behavior questionnaire. University of Massachusetts, 7 pp., http://people.umass.edu/~aizen/pdf/tpb.measurement.pdf.
Armitage
,
C. J.
, and
M.
Conner
,
2001
:
Efficacy of the theory of planned behaviour: A meta-analytic review
.
Br. J. Soc. Psychol.
,
40
,
471
499
, https://doi.org/10.1348/014466601164939.
Artikov
,
I.
, and Coauthors
,
2006
:
Understanding the influence of climate forecasts on farmer decisions as planned behavior
.
J. Appl. Meteor. Climatol.
,
45
,
1202
1214
, https://doi.org/10.1175/JAM2415.1.
Baumgart
,
L. A.
,
E. J.
Bass
,
B.
Philips
, and
K.
Kloesel
,
2006
:
Emergency management decision-making during severe weather
.
Proc. Hum. Factors Ergon. Soc. Annu. Meet.
,
50
,
381
385
, https://doi.org/10.1177/154193120605000336.
Cohen
,
T. R.
,
M. R.
Lindblad
,
J.-G.
Paik
, and
R. G.
Quercia
,
2009
:
Renting to owning: An exploration of the theory of planned behavior in the homeownership domain
.
Basic Appl. Soc. Psych.
,
31
,
376
389
, https://doi.org/10.1080/01973530903316971.
Demuth
,
J. L.
,
J. K.
Lazo
, and
R. E.
Morss
,
2011
:
Exploring variations in people’s sources, uses, and perceptions of weather forecasts
.
Wea. Climate Soc.
,
3
,
177
192
, https://doi.org/10.1175/2011WCAS1061.1.
Fagerlin
,
A.
,
B. J.
Zikmund-Fisher
,
P. A.
Ubel
,
A.
Jankovic
,
H. A.
Derry
, and
D. M.
Smith
,
2007
:
Measuring numeracy without a math test: Development of the subjective numeracy scale
.
Med. Decis. Making
,
27
,
672
680
, https://doi.org/10.1177/0272989X07304449.
Fischer
,
H. W.
, III
,
1996
:
What emergency management officials should know to enhance mitigation and effective disaster response
.
J. Contingencies Crisis Manage.
,
4
,
208
217
, https://doi.org/10.1111/j.1468-5973.1996.tb00095.x.
Hoss
,
F.
,
2014
: Uncertainty in river forecasts: Quantification and implications for decision-making in emergency management. Ph.D. dissertation, Carnegie Mellon University, 207 pp., http://repository.cmu.edu/dissertations/432/.
Hoss
,
F.
, and
P.
Fischbeck
,
2016
:
Increasing the value of uncertain weather and river forecasts for emergency managers
.
Bull. Amer. Meteor. Soc.
,
97
,
85
97
, https://doi.org/10.1175/BAMS-D-13-00275.1.
Hu
,
Q.
, and Coauthors
,
2006
:
Understanding farmers’ forecast use from their beliefs, values, social norms, and perceived obstacles
.
J. Appl. Meteor. Climatol.
,
45
,
1190
1201
, https://doi.org/10.1175/JAM2414.1.
Morss
,
R. E.
,
2010
:
Interactions among flood predictions, decisions, and outcomes: Synthesis of three cases
.
Nat. Hazards Rev.
,
11
,
83
96
, https://doi.org/10.1061/(ASCE)NH.1527-6996.0000011.
Morss
,
R. E.
, and
F. M.
Ralph
,
2007
:
Use of information by National Weather Service forecasters and emergency managers during CALJET and PACJET-2001
.
Wea. Forecasting
,
22
,
539
555
, https://doi.org/10.1175/WAF1001.1.
Morss
,
R. E.
, and
E.
Wahl
,
2007
:
An ethical analysis of hydrometeorological prediction and decision making: The case of the 1997 Red River flood
.
Environ. Hazards
,
7
,
342
352
, https://doi.org/10.1016/j.envhaz.2007.09.004.
Morss
,
R. E.
,
O. V.
Wilhelmi
,
M. W.
Downton
, and
E.
Gruntfest
,
2005
:
Flood risk, uncertainty, and scientific information for decision making: Lessons from an interdisciplinary project
.
Bull. Amer. Meteor. Soc.
,
86
,
1593
1601
, https://doi.org/10.1175/BAMS-86-11-1593.
Morss
,
R. E.
,
J. K.
Lazo
, and
J. L.
Demuth
,
2010
:
Examining the use of weather forecasts in decision scenarios: Results from a US survey with implications for uncertainty communication
.
Meteor. Appl.
,
17
,
149
162
, https://doi.org/10.1002/met.196.
Moyano Dı́az
,
E.
,
2002
:
Theory of planned behavior and pedestrians’ intentions to violate traffic regulations
.
Transp. Res.
,
5F
,
169
175
, https://doi.org/10.1016/S1369-8478(02)00015-3.
Paolacci
,
G.
,
J.
Chandler
, and
P. G.
Ipeirotis
,
2010
:
Running experiments on Amazon Mechanical Turk
.
Judgm. Decis. Making
,
5
,
411
419
.
Pielke
,
R. A.
,
1999
:
Who decides? Forecasts and responsibilities in the 1997 Red River flood
.
Appl. Behav. Sci. Rev.
,
7
,
83
101
, https://doi.org/10.1016/S1068-8595(00)80012-4.
Pielke
,
R. A.
, and
R. T.
Conant
,
2003
:
Best practices in prediction for decision-making: Lessons from the atmospheric and earth sciences
.
Ecology
,
84
,
1351
1358
, https://doi.org/10.1890/0012-9658(2003)084[1351:BPIPFD]2.0.CO;2.
Tversky
,
A.
, and
D.
Kahneman
,
1974
:
Judgment under uncertainty: Heuristics and biases
.
Science
,
185
,
1124
1131
, https://doi.org/10.1126/science.185.4157.1124.
Weaver
,
J.
,
L. C.
Harkabus
,
J.
Braun
,
S.
Miller
,
R.
Cox
,
J.
Griffith
, and
R. J.
Mazur
,
2014
:
An overview of a demographic study of united states emergency managers
.
Bull. Amer. Meteor. Soc.
,
95
,
199
203
, https://doi.org/10.1175/BAMS-D-12-00183.1.
Zikmund-Fisher
,
B. J.
,
D. M.
Smith
,
P. A.
Ubel
, and
A.
Fagerlin
,
2007
:
Validation of the subjective numeracy scale: Effects of low numeracy on comprehension of risk communications and utility elicitations
.
Med. Decis. Making
,
27
,
663
671
, https://doi.org/10.1177/0272989X07303824.

Footnotes

Supplemental information related to this paper is available at the Journals Online website: https://doi.org/10.1175/WCAS-D-16-0088.s1.

© 2018 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

1

When the error in probability of precipitation (PoP) forecasts was higher, people tended to consult it more often. The error in the maximum temperature forecast was not a significant predictor (Demuth et al. 2011).

2

The 12 states not represented are Connecticut, Delaware, Idaho, Maine, Maryland, Montana, New Hampshire, New Mexico, North Carolina, Rhode Island, Vermont, and West Virginia.

Supplemental Material