Thank you to Crystal Burghardt, Jennifer Boehnert, Cindy Halley-Gotway, and Taylor Trogdon for their assistance with this research and analysis; to Pat Callahan at ResearchExec for his extensive assistance with the survey design and implementation; and to Greg Guibert, Heather Lazrus, and three anonymous reviewers for helpful comments on the manuscript. Thank you also to the following NWS personnel for their assistance and input throughout this research effort: Bradley Akamine, Bob Bunge, Dennis Cain, Curtis Carey, Sam Contorno, Bob Glahn, Carl Gorski, Andy Horvitz, Eli Jacks, Ron Jones, Suzanne Lenihan, Mark Mitchell, Daniel Nietfeld, and Jennifer Sprague. This work was funded by the NWS Office of Science and Technology and Office of Climate, Water, and Weather Services under Award NA06NWS4670013. Portions of the authors' time were supported by NCAR's Collaborative Program on the Societal Impacts and Economic Benefits of Weather Information (SIP), which is funded by the National Science Foundation and the National Oceanic and Atmospheric Administration through the U.S. Weather Research Program under Award NA06OAR4310119. The views and opinions in this paper are those of the authors.
Agresti, A., , and Finlay B. , 2009: Statistical Methods for the Social Sciences. Prentice Hall, 609 pp.
Barnes, L. R., , Gruntfest E. C. , , Hayden M. H. , , Schultz D. M. , , and Benight C. , 2007: False alarms and close calls: A conceptual model of warning accuracy. Wea. Forecasting, 22, 1140–1147.
Coleman, T. A., , Knupp K. R. , , Spann J. , , Elliott J. B. , , and Peters B. E. , 2011: The history (and future) of tornado warning dissemination in the United States. Bull. Amer. Meteor. Soc., 92, 567–582.
Demuth, J. L., , Lazo J. K. , , and Morss R. E. , 2012a: Assessing and improving the NWS point-and-click web page forecast information. NCAR Tech. Note NCAR/TN-493+STR, 75 pp + appendixes. [Available online at http://nldr.library.ucar.edu/repository/collections/TECH-NOTE-000-000-000-859.]
Demuth, J. L., , Morss R. E. , , Morrow B. H. , , and Lazo J. K. , 2012b: Creation and communication of hurricane risk information. Bull. Amer. Meteor. Soc., 93, 1133–1145.
DHHS, 2006: Research-Based Web Design and Usability Guidelines. U.S. Dept. of Health and Human Services, 267 pp. [Available online at http://www.usability.gov/guidelines/guidelines_book.pdf.]
Hawkins, R. P., , and Daly J. , 1988: Cognition and communication. Advancing Communication Science: Merging Mass and Interpersonal Processes, R. P. Hawkins, J. M. Wiemann, and S. Pingree, Eds., Sage Publications, 191–223.
Hayes, A. F., 2005: Statistical Methods for Communication Science. Routledge, 516 pp.
Hillstrom, A. P., , and Chai Y. , 2006: Factors that guide or disrupt attentive visual processing. Comput. Human Behav., 22, 648–656.
Lazo, J. K., , Morss R. E. , , and Demuth J. L. , 2009: 300 billion served: Sources, perceptions, uses, and values of weather forecasts. Bull. Amer. Meteor. Soc., 90, 785–798.
Lundgren, R. E., , and McMakin A. H. , 2009: Risk Communication: A Handbook for Communicating Environmental, Safety, and Health Risks. John Wiley and Sons, 362 pp.
Mayer, R. E., 2005: Cognitive theory of multimedia learning. The Cambridge Handbook of Multimedia Learning, R. E. Mayer, Ed., Cambridge University Press, 31–48.
Mileti, D. S., , and Sorensen J. H. , 1990: Communication of emergency public warnings: A social science perspective and state-of-the-art assessment. Oak Ridge National Laboratory Manuscript ORNL-6609. [Available online at http://emc.ed.ornl.gov/publications/PDF/CommunicationFinal.pdf.]
NOAA, 2011: NWS Central Region service assessment: Joplin, Missouri, tornado—May 22, 2011. National Oceanic and Atmospheric Administration, 34 pp. [Available online at http://www.nws.noaa.gov/om/assessments/pdfs/Joplin_tornado.pdf.]
NOAA, cited 2013: NDFD spatial reference system. [Available online at http://graphical.weather.gov/docs/ndfdSRS.htm.]
Renn, O., 2008: Risk perception: Review of psychological, social and cultural factors of risk perception. Risk Governance: Coping with Uncertainty in a Complex World, O. Renn, Ed., Earthscan, 98–148.
Scharfenberg, K., , Walawender B. P. , , and Akamine B. , 2012: Public survey results for a prototype NWS web-based watch/warning/advisory map. Preprints, Seventh Symp. on Policy and Socio-Economic Research, New Orleans, LA, Amer. Meteor. Soc., 10.2. [Available online at https://ams.confex.com/ams/92Annual/webprogram/Paper197329.html.]
Shadish, W. R., , Cook T. D. , , and Campbell D. T. , 2001: Experimental and Quasi-Experimental Design for Generalized Causal Inference. Wadsworth Publishing, 656 pp.
Smyth, J. D., , and Pearson J. E. , 2011: Internet survey methods: A review of strengths, weaknesses, and innovations. Social and Behavioral Research and the Internet: Advances in Applied Methods and Research Strategies, M. Das, P. Ester, and L. Kaczmirek, Eds., Routledge, 11–44.
Sue, V. M., , and Ritter L. A. , 2007: Conducting Online Surveys. Sage Publications, 208 pp.
Sutter, D., , and Erickson S. , 2010: The time cost of tornado warnings and the savings with storm-based warnings. Wea. Climate Soc., 2, 103–112.
Zimmerman, D. E., , and Akerelrea C. , 2003. Usability testing: An evaluation process for Internet communications. Internet Encyclopedia, H. Bidgoli, Ed., John Wiley, 512–524.
Zimmerman, D. E., and Coauthors, 2010: Evaluation and usability testing of two National Weather Service web sites: Denver/Boulder Weather Forecast Office web page and Fort Collins point-and-click forecast web page. Dept. of Journalism and Technical Communication Rep., Colorado State University, 68 pp.
The research pertained to the point-and-click information content and layout as it was presented for over a decade up until 2 July 2012 when NWS fielded a redesigned PnC forecast web page. The redesign was informed in part by findings from the initial phase of the research effort.
The PnC forecast information is populated by the Graphical Forecast Editor (GFE) forecast grids that are updated at least twice daily by forecasters in each NWS Weather Forecast Office. Currently, there is a unique PnC forecast for every 2.5-km2 grid in the continental United States, Hawaii, and Guam; for every 1.25-km2 grid in Puerto Rico; and at least for every 6-km2 grid in Alaska (NOAA 2013).
Hazardous weather was denoted in this way on the PnC web page at the time of the study and also on the current redesigned web page.
This is one way we attempted to minimize threats to internal validity. Internal validity is the extent to which a causal relationship exists between independent and dependent variables. Internal validity of a study is threatened when an extraneous variable—for example, in this case, the real forecast location of Billings, Montana—unintentionally influences study participants, confounding the effects of the independent variable (Shadish et al. 2001).
The NWS has prototyped and obtained public feedback on a new web-based watch–warning–advisory map where warnings are denoted in red, watches in orange, and advisories in yellow (Scharfenberg et al. 2012). This color scheme is commensurate with typical color associations of U.S. adults (Sue and Ritter 2007); therefore, we emulated it in the graphical attributes added to the experimental forecast presentations.
The NWS reviewers were selected to represent a range of expertise and perspectives regarding the PnC web page. They included NWS Headquarters employees who manage the web-based forecast policies, NWS technical staff who manage the PnC web page programming, and NWS forecasters whose forecasts populate the PnC.
Ranges are across the survey samples.
As part of our data analysis, we conducted chi-squared (χ2) tests of independence statistical tests. Used for categorical data, they test whether the conditional distributions of the dependent variable (e.g., threat identification) are identical across the experimental forecast presentations in each weather scenario. Independence means that the probability of any particular response is the same regardless of the experimental presentation. If the conditional distributions vary by forecast, they are dependent; that is, they differ by experimental presentation (Agresti and Finlay 2009). We adopted a stringent p ≤ 0.01 level when interpreting the χ2 statistical results to minimize the chance of incorrectly concluding that there are differences among the experimental presentations for a given dependent variable (i.e., a type I error). With each χ2 statistical result, we also report the effect size (as Cramer's V), which is a measure of the strength of association between the independent variable (i.e., experimental presentations) and dependent variable. Values of Cramer's V range from 0 to 1, with larger values representing stronger associations (Hayes 2005).
Severe thunderstorm warning scenario start-time options were “It has already started,” “6 PM on Wednesday,” and “Sometime tonight”; end-time options were “It has already ended,” “6 PM on Wednesday,” and “Sometime tonight.” Flood watch scenario start-time options were “It has already started,” “Sometime Thursday,” and “6 PM on Saturday”; end-time options were “It has already ended,” “Sometime Thursday,” and “6 PM on Saturday.”
Severe thunderstorm warning scenario start-time options were “It has already started (i.e., now),” “6 PM today,” and “Sometime tonight”; and end-time options were “It has already ended,” “6 PM today,” and “Sometime tonight.” Flood watch scenario start-time options were “It has already started (i.e., now),” “6 AM on Thursday,” and “6 PM on Saturday”; and end-time options were “It has already ended,” 6 AM on Thursday,” and “6 PM on Saturday.”