The quasi experiment of deviations from normal temperatures shows how local temperature conditions bias selected survey results. Responses in eight CBS News surveys from 2001 to 2007 change with the weather, with unseasonable temperatures reducing concern about climate change and unusually warm temperatures increasing presidential approval. Unusual temperatures also influence who answers surveys; wealthier respondents are overrepresented in warmer conditions. These results jointly suggest that surveys are at risk for temperature-induced response bias. Weighting-based methods can account for survey results’ temperature-induced differences in samples.
Temperature conditions may induce selection bias in survey research. People adjust their schedules to avoid heat or work indoors when it is cold; varying temperature conditions could thus alter who responds to surveys—especially if income groups’ responses to temperature differ (say, because of differential access to climate control). Outdoor temperatures could then lead to sample effects, biasing respondents’ income distribution. At the same time, weather affects people’s moods and dispositions so that varying temperature conditions can change respondents’ answers to survey questions by opinion effects altering their perceptions of the world; for example, if unusually cold temperatures dispirit people, they may be likelier to express disapproval with politicians’ performance (e.g., Cohen 2011). This poses problems: when a stimulus produces both sample and opinion effects, determining which mechanism induced observed outcomes is difficult. Without correcting for temperature (and perhaps other weather conditions), scholars risk drawing inaccurate or unreliable conclusions from virtually all surveys.
We exploit temperatures’ quasi-random variation to investigate opinion and sample effects on survey response. We first document significant sample effects as local temperatures distort respondent demographics; respondents tend to be richer during unusually cold temperatures. We then correct for these sample effects with weights, which changes the interpretation of weather-based effects on attitudes toward global warming and presidential approval. This implies that accurately measuring public opinion requires adjusting standard survey results with conditioning strategies, such as weights, propensity score matching, or regression-based controls relating to weather.
2. Temperature, public opinion, and surveys
Though many components of weather (e.g., precipitation or windiness) can affect behavior, we focus on deviations from normal temperature. People have stronger expectations about temperature than about the likelihood of sunlight or rainfall, both because temperature is more predictable and because probabilities are cognitively difficult. This strong expectation relates to acclimatization: people respond to the same absolute temperature with t-shirts in January but with jackets in August, showing that deviations from seasonal temperature norms represent a stochastic process with a clear perceptual baseline. Temperature variation also has clear connections to contentious policy issues, as discussed below. Finally, temperature deviations conveniently proxy for other weather-based causes; for instance, cloud cover may change the likelihood of abnormally high temperatures.
Sample effects occur when social groups’ divergent reactions to temperature change their relative likelihoods of responding to surveys. In particular, temperature can affect different income groups’ propensities to be home to answer survey phone calls so that unusual temperature conditions will bias survey samples’ income distributions. Working class professions, for example, construction and agriculture, more often require outdoor work likelier to be postponed in extreme temperature conditions, leaving poorer people likelier to be available to respond to telephone surveys. Richer people are also less likely to use public transit, making them more likely to venture from home in unusual temperatures (errand running can be postponed when it involves waiting in the cold at the bus stop). Conversely, richer people are likelier to have climate control such as air conditioning in their homes, while poorer people may leave in search of comfortable settings during unusual temperature conditions.
To investigate temperature’s opinion effects, we first consider opinions about global warming. Though science only weakly links climate change with daily temperature, unusually warm temperatures are often seen as increasing worry about global warming (Egan and Mullin 2012). Hence, 1988’s unusually hot summer brought global warming to the public consciousness (Leiserowitz 2005, p. 1435), while official Russian policy and rhetoric seem to have changed in response to 2010’s devastating heat wave; previous climate change skepticism abruptly reversed (Gillis 2010). Other effects are possible, for example, if heightened media attention to climate change during unusual temperatures (heat waves or cold snaps) prompts a backlash that reduces, rather than stimulates, acceptance of the idea that humans affect climate. Regardless, survey respondents may react to atypical temperatures by changing their concern about global warming.
Moreover, temperature’s effect on attitudes extends beyond climate policy into broad, central questions of politics. Warmer temperatures typically improve most people’s moods (albeit less so in summer; Keller et al. 2005). Mood in turn shapes judgments of political circumstances; more content individuals approve more of incumbent politicians, even if the good feelings’ source was nonpolitical (Healy et al. 2010). This means temperatures may affect outcomes such as presidential approval, our second substantive question of interest. The above logic implies that higher-than-normal temperatures will associate with greater likelihood of respondents’ approving of the president’s performance, a conclusion supported by occasional previous research (Cohen 2011).
These two different potential implications of temperature conditions—opinion and sample effects—complicate survey interpretation by occurring simultaneously. Observing higher proportions of Republicans during heat waves, for example, is ambiguous; did it result from differential survey participation among would be Democratic and Republican respondents? Or are essentially similar respondents reacting to the temperature in ways that made their partisan orientation more Republican? Attitudes toward individual issues are even more likely to be strongly influenced by external conditions than are broad, blunt factors such as preferred party.
Thus, our empirical analysis has two distinct goals. We first consider whether survey samples vary with temperature conditions. We then consider whether temperatures influence attitudes toward global warming and presidential performance, building temperature-adjusted survey weights to check for sensitivity to temperature-induced sample bias. Even partial correction for temperature-related sample differences changes the inferences drawn from surveys, raising general questions for survey-based social science.
3. Methods and data
Studying these questions requires a dataset of surveys administered from all seasons (to ensure variation in temperatures) with information about respondents’ location and questions on climate change and presidential approval. We accordingly used the broadest available dataset of surveys containing consistent questions on these issues: eight random digit–dialed CBS News/New York Times surveys from June 2001 to December 2007. Each survey occurred in a different month, producing wide seasonal variation among the 7631 pooled observations from across the contiguous United States.
The key independent variable, local temperature conditions, uses data from the NOAA National Climatic Data Center (NCDC). Some areas lack weather stations, while other areas have many. To measure temperature consistently, we thus spatially interpolate (krige) temperatures for each county seat for both the day of the survey (e.g., 20 January 2007) and for the 1971–2000 normal temperatures for the date (20 January). The difference between these two records then serves as our measure of interest: the extent to which respondents’ temperature conditions differed from normal. Unusually hot days for a location at the time of year have large positive values, unusually cold days have large negative values, and near-normal days have values near zero. (Alternatively, we assign respondents to the survey day’s conditions at the weather station nearest the respondent’s county seat, thus without spatial interpolation. This produces very similar results to those described.) Examining the difference between two interpolations should net out anomalies such as urban heat bubbles.
The exogeneity to social factors of deviations from normal temperature creates a quasi experiment, albeit with treatments applied to county days, not individuals. Local temperature conditions on any particular day are essentially independent of social factors such as income or presidential approval. The randomization is, however, not perfect, since some areas’ and seasons’ temperatures vary more than others’ temperatures: weather is less variable in Los Angeles in September than in Fargo in January. Also, cell phone uptake increased over time, altering the demographics for landline surveys like those used here (Lavrakas et al. 2007). Consequently, regression analyses below include unreported dummy variables for each state and survey.
We present two sets of analyses. The first examines sample effects: whether temperature conditions change sampled survey demographics, focusing on temperature’s relation to respondents’ income. The second examines how controlling for these sample effects changes the predicted effect of weather on opinions concerning on climate change and presidential approval.
4. Results: Temperature and survey samples
We begin by investigating potential temperature biases in response rates across different income groups. For these analyses, the dependent variable is respondent family income, self-reported on an ordinal, five-point scale. The key independent variable is temperature conditions at the most specific geographical location available for the respondent, the county. To allow for nonlinear effects of temperature if, for example, people systematically stay home during both unusually warm and cold weather, models include both the deviation from normal temperatures (measured as described in the previous section) and the square of this deviation; dividing the temperatures into indicator variables for various ranges produces similar results, suggesting that negative and positive temperature deviations have similar effects.
Even without explicitly controlling for temperature effects, existing survey-weighting methods might implicitly control for temperature. This would still be an imperfect solution if outcome or control variables also correlate with temperature conditions, but it may alleviate concerns about sample effects. To see whether weighing ameliorates the relationship between temperatures and survey response, the sample effect analyses include models reporting results with the CBS surveys’ calculated final weights as well as models excluding such weights.
Table 1 reports the resulting regression coefficients, with the quasi randomness of temperature deviations reducing the need for control variables besides dummy variables for each survey and state. Columns I and II provide the main results, with ordered logistic regressions looking across all five income categories. To examine whether observed effects derive purely from the behavior of poor or rich income groups, columns III through VI run binary logistic regressions predicting membership in one or the other extreme income categories.
In columns I and II, higher temperature deviations associate with higher income samples. This relationship may diminish when temperatures are well above normal; the highest predicted income value occurs when temperatures are roughly 4°C above normal. The quadratic variable’s p statistic rises to 0.059 in column II’s specification including survey weights though, so temperature effect’s nonlinearity appears possibly fragile. With the weighted data, standard errors generally increase considerably, reducing the statistical significance of coefficients. Most coefficients are nevertheless comparable in size to those of the unweighted regression, as would be expected of weights constructed independently of temperature conditions.
The last four columns of Table 1 show more directly temperature’s effects on the probability of the wealthy (columns III and IV) and poor (columns V and VI) being in the survey sample. Higher temperature deviations generally associate with more likelihood of respondents being at either end of the income scale; all of the coefficients on the linear effect of temperature are positive (though the quadratic term again indicates that the relationship is weaker in unusually warm temperature ranges). However, the coefficients on temperature deviation for wealthy respondents (columns III and IV) are consistently larger than those for poorer respondents (columns V and VI), suggesting that temperature variation has more substantive significance for the survey participation of the rich than of the poor. The supplemental material contains further details on how temperature deviations affect survey sample compositions.
5. Results: Postsurvey sampling weights
Having documented sample selection effects, we turn now to investigating possible opinion effects and whether temperature-adjusted survey weights affect survey results. To implement this approach with our data, we estimated populations in each income–temperature group by finding the distribution of temperature deviations in each state in each month by measuring the share of readings for each weather station that were 3°C (or more) above the usual temperature for that date location, along with the share of readings that were 3°C (or more) below the usual temperature. These proportions were averaged for all weather stations in a state to estimate the distribution of temperatures in a state; how likely is it that someone in the state will experience an unusually warm (more than 3°C above normal, though other category boundaries preserve the substance of the results) day in a given month? Combining this information with data from the U.S. Census’s American Community Survey concerning county income distributions produced a population estimate for each of the survey’s primary sampling units (PSUs) for each income category and temperature level.
For this paper, comparing these population estimates to the survey income–temperature distribution identified the sample’s under- and overrepresented groups and thus who merited increased or reduced weighting to balance the sample. These new weights could then be reapplied to models predicting global warming and presidential-approval attitudes. Columns III and VI of Table 2 present results using temperature-adjusted weights alongside other columns using no weights or standard, temperature-disregarding weights. Accounting for temperature changes predicted effects: when using temperature-adjusted weights, the estimated coefficients tend to be smaller, while the uncertainty surrounding the estimates is larger (as is typical with weighted results). This preliminarily suggests that observed variation of opinion under different temperature conditions may reflect differences in sample composition rather than actual opinion effects.
Table 2’s temperature-based weights have the caveat that they might miss some survey sampling effects. For example, our weighting mechanism excludes age, which figures substantially in the polls’ provided weights; without county-level information from the Census data about how frequently different age groups are in the various income groups experiencing unusual temperature conditions, we cannot calculate a weight incorporating these response effects. More generally, little information is available about demographic distributions under different temperature conditions. If younger, more mobile populations have tended to seek out and move to places with a mild, invariant climate, the demographics of people experiencing very warm and very cold temperatures will differ from the general population’s demographics. This is especially important because these age effects plausibly interact with income: young people who can afford to relocate to climatically attractive locations may be relatively well-off. Detailed information about different demographic groups’ propensities to face varying weather conditions would thus have immense value for survey research.
Temperature conditions may induce some bias in survey responses; surveys conducted in unusually cold and (sometimes) warm weather tend to have wealthier samples, even when using traditional survey weights. This has important implications for those contemplating weather’s social consequences and for public opinion researchers in general. Survey takers ought to account for temperature by collecting information on respondents’ weather—or at least on geography from which weather information can be derived. Such data would help scholars identify temperature-related effects on sampling and opinions; this is especially important as climate change becomes more pronounced and weather-related issues become increasingly salient for a wide range of social research. Survey practitioners should also check temperature as a factor in developing their standard survey weights made available for public use. This may suggest other approaches to reducing the effect of weather on results, such as extending sampling periods so that the survey duration is likely to capture a more representative range of local temperatures.
These results suggest several avenues for future research. For what issues are opinions dependent on local temperatures? How do other temperature conditions, such as precipitation, alter respondents’ representativeness and political opinions? Does temperature differentially affect other subpopulations, such as those caring for children, living in different regions, or in rural areas? Considering these factors may pinpoint the particular causal mechanisms leading different groups to react differently to temperature. The answers will help sharpen inferences about how weather shapes mood, opinion, and society.
The authors thank Henry Huntington and the anonymous reviewers for helpful comments.