1. Introduction
Understanding the future climate and its impacts is critical for the population and is one of many concerns associated with climate change, as the World Health Organization estimates climate change could cause at least 250 000 deaths per year from 2030 to 2050 (Haines and Ebi 2019). However, to much of the public, climate change is uncertain and very ambiguous, as only 57% of Americans believe that global warming is mostly caused by human activities (Marlon et al. 2022; Nisbet 2009). As such, communication in this field has become urgent, but it is challenging (Somerville and Hassol 2011). One way to communicate climate change is through images and graphs, but visuals involving climate change need to be carefully considered and implemented in order to be most effective with their intended audiences (Atkins and McNeal 2018; Corner et al. 2018; Harold et al. 2016).
One target audience that requires effectively communicated, accessible climate information is managers and policy makers, as they must make decisions about the impacts of climate change on the systems they manage. Decision support systems (DSSs) are tools that can be used to bridge the information gap for managers and policy makers and can specifically aid with communication about climate change. “Decision support provides a link between decision making, scientific information, and analytical tools” and can take many forms, such as documents or software tools (Pyke et al. 2007). DSSs have information that helps users, specifically stakeholders in a particular field, understand facts and make critical decisions. Climate change has been, and is going to continue to be, at the forefront of concern for decision-makers, especially those involved with managing and protecting wildlife.
Climate change has already had a significant impact on many endangered species (Hansen et al. 2006; Reidmiller et al. 2018). Shifts in weather patterns including extreme temperatures and precipitation can affect a species’ ongoing survival within its native range, and understanding these changes can inform prudent management decisions. Under the Endangered Species Act (ESA), the U.S. Fish and Wildlife Service (USFWS) is required to evaluate the status of at-risk plants and animals in the United States. This evaluation is formally conducted through a species status assessment (SSA), which is a rigorous risk-assessment document developed by the USFWS and prepared for at-risk species to help inform a range of management decisions under the ESA. SSAs tend to focus on the representation, redundancy, and resiliency of a specific endangered species (https://www.fws.gov/project/species-status-assessment). DSS tools can assist species managers in their assessment and decision-making by providing the most up-to-date climate science information while considering known impacts to key regional species, such as temperature thresholds above or below which their behavior is changed. A framework specifically developed for the USFWS in the form of a DSS has been identified as being needed to assist proper management of critical species by linking decision-making, scientific information, and analytical tools (Pyke et al. 2007).
To help USFWS biologists incorporate climate information into SSAs, the State Climate Office of North Carolina (SCONC) outlined a framework for the Climate Analysis and Visualization for the Assessment of Species Status (CAnVAS) website. This DSS was based on a previous tool created by the SCONC called the Pine Integrated Network: Education, Mitigation, and Adaptation Project (PINEMAP) DSS, which was designed for professional loblolly pine (Pinus taeda) foresters who likewise faced future climate-affected management decisions (Aldridge et al. 2016; Davis et al. 2020; Maudlin et al. 2020). The climate information provided in CAnVAS includes projected future changes in key climate variables across 20- or 30-yr time intervals, along with a measure of model spread to best illustrate the range of possible conditions for a given location. Based on lessons learned about interface design and visualization from the PINEMAP DSS, and informed by an initial consultation with regional biology experts about the technical needs of SSAs, the SCONC created mock-ups of the pilot version of CAnVAS used in this study.
This development approach intentionally used “prompted transitions from scientist to stakeholder-driven or collaborative approaches to climate science” (VanderMolen et al. 2020). Known as coproduction, this involves science makers (researchers) collaborating with science users (managers/stakeholders) to create a final product that can be successfully employed in decision-making to produce usable climate science knowledge (Meadow et al. 2015). Although coproduction approaches have been recently used in actionable science, science that can be used for making future decisions, the products that result from these efforts are often not rigorously evaluated for their usability or use with the intended stakeholders (Durose et al. 2014; Wall et al. 2017). By providing information from initial trials of this research, SCONC will adjust CAnVAS to be more user friendly, which builds upon the process of coproduction and contributes to the uniqueness of this study. The near-term contribution of this study is that CAnVAS will be refined further for future stakeholder use after initial pilot testing. The broader contribution of this work is to provide an example of how to conduct a rigorous evaluative process during the development of a coproduced product that is ultimately intended to be used by a specific stakeholder.
Okan et al. (2016) explained that graphical displays (bar graphs, scatterplots, etc.) are effective for DSSs if they aid in comprehension of information. Previous research has tested differences between comprehension of bar and line graphs through a mixed design and research on evaluating graph comprehension and pattern recognition through eye tracking (Carpenter and Shah 1998; Shah and Freedman 2011). Previous research has also identified that “rainbow” color ramps are not qualified enough for displaying information due to user confusion and complicating data (Moreland 2009; Turton et al. 2017). When choosing colors for interpretation, it is essential to consider the color and background on which the color ramp is being displayed (Schloss et al. 2019). This information provides justification for this current study.
2. Objectives
This study aims to understand the differences in comprehension of quantitative climate information that will be displayed on DSSs. The A/B study design uses limited independent variables to enable a direct comparison of mock-ups of the CAnVAS tool with the following differences: (i) how information is displayed on a graph, through either a faded-bar or boxplot layout and (ii) the color ramps used for regional map displays, either spanning from white to purple or from blue to brown, indicating low to high values. The overarching goal of the larger project is to fill the assessment gap in coproduced research by designing a web-based DSS for species managers and evaluating its usability when applied to the construction of SSAs. The research questions guiding this project include the following: (i) How do novice users engage with different versions of CAnVAS? (ii) How can the usability of CAnVAS be improved?
These questions were answered through a mixed-methods approach using eye tracking and interviews with novice users (undergraduates). Previous usability testing has been done on various types of DSS; however, eye tracking with participants was not deployed as a method for usability assessment in those studies as in this current study (Oztekin 2011; Arciniegas et al. 2013; Zhang et al. 2015; Han et al. 2018). Eye tracking allows for correlating “eye-tracking metrics used and theoretical constructs of interest” and assessing usability through tracking participants’ eye movements and showing where and for how long participants looked at certain areas (Ehmke and Wilson 2007; Haesner et al. 2018; King et al. 2019). In this and other studies, eye tracking has measured the usability of web-based tools by looking at three metrics: efficiency, effectiveness, and satisfaction (Courtney 2019, p. 7; Creager and Gillan 2016; Maudlin et al. 2020). Efficiency was measured by how long it took a participant to choose an answer based on post–DSS viewing questions provided during the eye-tracking experiment. Effectiveness was measured by participant performance scores on post–DSS viewing questions. Finally, satisfaction with CAnVAS was measured by the overall participant perceived usefulness of the DSS through retrospective interviews with participants.
3. Methods
a. Development of CAnVAS 1.0
For the development of the pilot version of the CAnVAS DSS, SCONC met with a committee of stakeholders associated with the USFWS to assess needs for this DSS, which was an important step in the development process (McNie 2013). These meetings allowed for the process of coproduction to be a primary focus of this research and made it unique in that end users provided input from the very beginning. This pilot iteration of CAnVAS was a still-image layout rather than a “live” navigational website. The codevelopment process used in the PINEMAP DSS suggested that first developing such mock-ups provides a suitable platform for initial evaluation, and any major changes or functionality concerns can then be addressed in the more time-consuming web-design step (Davis et al. 2020). These mock-ups were designed to show potential climate changes for one endangered species—the gopher tortoise. Similar to Hegarty et al. (2010), an A/B study design was developed where two versions of CAnVAS were made to allow for testing of specific design features. Two visual design differences were examined between version A and B—how data were displayed in the climate-projection portion (i.e., faded bar or boxplot) and the climate snapshot color ramps (i.e., from white to purple or from blue to brown). These design features were chosen for manipulation, as previous research points to testing differences between graphical designs within DSSs to ensure effective communication and determining the best sequential color ramps to use to depict climate information (Carpenter and Shah 1998; Moreland 2009; Okan et al. 2016; Schloss et al. 2019; Shah and Freedman 2011; Turton et al. 2017).
b. Eye tracking and interviews
Eye tracking, combined with postinterviews, is an established method for improving and designing web-based applications (Davis et al. 2020; Manson et al. 2012; Maudlin et al. 2020). In this study, an undergraduate population was tested through eye tracking and interviews. Qualitative and quantitative results yielded a set of key usability improvements and recommendations for the CAnVAS web interface. While the undergraduate population is not stakeholders in the species status assessment process, they are an appropriate sample population in a pilot study for research aimed to evaluate data-display elements and inform design changes for the next version of the tool. The next version of the DSS tool will then be tested with stakeholders in the USFWS. The work was conducted during the height of the COVID pandemic, so access to federal employees was difficult, and undergraduates could be accessed more safely and easily by the researchers. Before recruitment of undergraduate participants began, researchers reviewed the CAnVAS mock-ups with SCONC developers to outline the study procedures, including tasks and questions to guide users through the interface and assess their understanding. Minor adjustments were made to avoid potential confusion in answer choices and make sure all elements were clear and visible on the eye-tracking computers. The two mock-up versions of CAnVAS that were used for this study are shown in Fig. 1. We use the terminology “boxplot” to indicate the top of Fig. 1 and “faded bar” to indicate the bottom of Fig. 1.
The (top) boxplot and (bottom) faded-bar versions of CAnVAS tested with undergraduates.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0010.1
In these mock-ups, two pieces of information were preselected: the species (gopher tortoise) and location (Auburn, Alabama). These will be selectable by users in the final interface, but, since the focus of this study was on comparing data visualizations, that selection process was excluded. On all mock-up versions, a section titled “climate projections” displayed a graph of a sample of key climate variables (average maximum temperature, average precipitation, and maximum temperature occurrence above a 90°F threshold). On one mock-up, graphs used the boxplot format, while on the other, graphs used faded bars. A bottom section titled “climate snapshots” showed three regional maps of one parameter (maximum temperature occurrence) for only the 2010–39 time interval. On the boxplot version, data on the map were shaded using the white-to-purple color ramp, while on the faded-bar version, the map used the blue-to-brown color ramp. All other elements of the interface and the study (e.g., axis labels, legends, and questions asked of users) remained consistent between versions.
Note that the data shown in the charts and maps on these mock-ups did not use the actual climate-projection grids that will be implemented in the working CAnVAS website. Rather, the maps used generic climate data with contrasting values across the Southeast in order to make answering the task-related questions easier (e.g., where the color and its corresponding value for Auburn were nonambiguous). Similarly, the graph data were organized such that each time period’s bar had a clearly identifiable range based on the axis labels. A similar approach of using “dummy” data on initial iterations of tool mock-ups was used for the PINEMAP DSS development (Davis et al. 2020) since the goals of this stage of the product-development process were identifying visualization improvements rather than critically analyzing the future projections.
Once agreeing to be part of the study, participants came to the laboratory and sat at the desktop computer at approximately 65 cm from the eye tracker, which was located at the bottom of the computer monitor, and were calibrated with the TX-300 Tobii eye tracker, allowing for precise measurements of the eye tracker. The calibration process required the participant to follow red dots across the screen, and calibrations of >70% weighted gaze samples were allowed to continue to the main study. Data were collected within the Tobii Studio 3.4.8 software.
Participants were randomly assigned to the two different versions of CAnVAS (i.e., boxplot or faded bar). Both study groups viewed the two free exploration components of CAnVAS for 1 min and 15 s to allow participants to get acquainted with the CAnVAS interface before answering questions. One free exploration occurred before question 1 and the other occurred before question 11. Participants answered 15 questions about CAnVAS, while able to look at the CAnVAS interface, each with 6 different answer choices. There was no time limit on how long a participant could spend answering a question. These 15 questions were divided into 3 different tasks that included an equal number of 5 questions each. Each task included questions pertaining to specific portions of the CAnVAS interface (climate snapshot, occurrence of maximum temperature, climate projection). Table 1 shows the different tasks from the study. The participants’ eye movements were tracked the entire time they spent answering the question. Once participants were done with the eye-tracking portion of the study, approximately 15-min long retrospective interviews were conducted, and audio was recorded. During these interviews, participants viewed their own eye-tracking data replay to understand exactly what their eye movements looked like and to guide potential questions asked to participants.
Each task asked during the research study on the CAnVAS interface.
Overall, 39 undergraduate participants signed up to participate and were involved in the eye-tracking portion of this research. However, two participants’ data were discarded because they did not reach the requirement of at least 70% weighted gaze samples during the study, which left 37 participants for data analysis. With random assignment, there were 19 participants in the faded-bar version and 18 participants in the boxplot version of CAnVAS.
After all participants completed the eye-tracking portion and interviews, areas of interest (AOIs) were created for each CAnVAS mock-up version. AOIs are drawn through the Tobii Studio software on a still image and are used to determine quantitative metrics of participants’ eye movements across the particular section (e.g., time to first fixation). AOIs remained consistent across both versions of the study, meaning that AOIs were the same size and in the same location for both versions. For some questions, smaller AOIs were drawn within larger AOIs in order to get an accurate representation of whether participants were looking at the appropriate portion of a map or graph when trying to find the correct answer to the question. An example of AOIs for one question of CAnVAS is located in Fig. 2.
Examples of AOIs from task 3 of CAnVAS.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0010.1
In this research study, AOIs were used to measure the following eye-tracking metrics: total fixation duration (TFD), time to first fixation (TFF), and time to first mouse click (TFMC). TFF is how long it took for a participant, from the start of the question, to fixate on a certain AOI. TFD is how long the participant spent fixated on a certain AOI. Participants could fixate on a certain AOI more than once by coming back and forth to that AOI. TFD and TFF were normalized to TFMC. TFMC was used as a proxy for how long a participant spent on the particular question, since at the TFMC in the answer AOI, the participant would have selected an answer. Normalizing this data was crucial to get an accurate representation of how long a participant spent in AOIs measured relative to the time spent on the web page.
4. Results and discussion
a. Eye-tracking analysis
A bulk analysis was conducted for TFD, TFF, and TFMC. These eye-tracking metrics were summed, per group, across all tasks. These metrics were not normalized, since this was an initial bulk analysis, and the goal was to understand if there were any obvious differences between groups before implementing a finer analysis. These totals were then analyzed through an independent samples t test to measure significance and calculate Cohen’s d, a measure of effect size. After the bulk analysis, there was no statistical significance for the eye-tracking metrics, but there was a large (where large > |0.5| and will be referred to as such throughout the rest of the results) effect size for TFD. Despite the lack of statistical significance, the authors consider that Cohen’s d and t-test results are independent measures. As such, effect size is chosen as an evaluation metric, because it indicates a potential difference between the faded-bar and the boxplot groups. The statistical analysis is dependent on sample size and is a limitation of this study.
For a finer analysis, TFF was measured on the correct-location AOI and used to understand how long it took participants from both groups to look at the correct spot on CAnVAS. Correct location, in this study, is defined as where the participant would have had to look to retrieve the correct answer to the question. To have a robust analysis, effect size was calculated for each individual correct location on every task for TFF. There were five correct locations that had large effect sizes (where large > |0.5|): tasks 2.2, 3.2 B, 3.3, 3.4, and 3.5. Note that there were some instances in which participants did not look at the correct answer to register a measurement for TFF.
The next eye-tracking metric that was analyzed was TFD for correct locations on each question. Analyzing TFD for correct locations allowed the researchers to understand how long users focused on the correct location. Once again, the same correct locations were used for this analysis as previously identified. Per the bulk analysis, there was no statistical significance between groups. However, there was a large effect size for the bulk analysis of TFD for correct locations in tasks 2.4, 3.3, and 3.4 B, 2.2, and 3.4.
The one task that had a large effect size with the correct location for both TFF and TFD was task 3.3. This task asked for the participant to find the midrange of the model projections for the occurrence of maximum temperature greater than 90°F during the 2010–39 period in Auburn. This required the participant to look at the middle panel in the climate snapshots on the bottom half of CAnVAS and focus on Auburn to see what value correlates with the color in the legend on the left, which is indicated by a green box in Figs. 3 and 4. Participants could have looked at an alternative location in the climate projections to find the answer, which is indicated by a red box in Figs. 3 and 4, but the main goal was to have the participant look at the climate snapshots (green box) during all of task 3. To properly understand the differences between the two groups, heat maps of both versions of CAnVAS are provided in Fig. 3. One participant from each version of CAnVAS is shown in the example gaze plot included in Fig. 4. Gaze plots show the path taken by participants when completing the task in each group. These two participants were chosen for a representation of gaze plots because they spent almost the same amount of time, approximately 40 s, on this task, even though they were provided different versions. In the heat maps shown in Fig. 3 and gaze plots shown in Fig. 4, it is evident that overall the boxplot participants looked at the correct location in the climate snapshot rather than the alternative location in the climate projections. The opposite is true for the faded-bar participants, as they used the climate projections rather than the climate snapshots to answer the question.
(top) Boxplot and (bottom) faded-bar heat maps for task 3.3 with all participants’ data, where the green-outlined box indicates the correct location for the question, and the red-outlined box indicates the incorrect location for the question.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0010.1
(top) Boxplot and (bottom) faded-bar gaze plots of task 3.3 for two participants, one from each version of CAnVAS, where the green-outlined box indicates the correct location for the question, and the red-outlined box indicates the incorrect location for the question.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0010.1
b. Usability: Efficiency, effectiveness, and satisfaction
Efficiency was measured using the TFMC metric (Table 2). TFMC was determined for all 15 questions across both of the treatment groups. TFMC was summed to test for statistical significance across groups through an independent samples t test. There was no statistical significance nor large effect size between groups for total TFMC using bulk analysis. Thus, a finer analysis was done on TFMC through an independent samples t test with each individual task. There was also no statistical significance for the TFMC for individual tasks, but averages of TFMC for each individual task between groups did vary.
Analysis of TFMC on the answer AOI for both versions of CAnVAS.
Even though there was no statistical significance, there were differences in the averages of how long it took for participants to choose an answer to the question (TFMC) between the two groups. Note that with the minimal number of participants, there was no statistical significance; however, it is believed that with a larger sample size, statistical significance could have been reached. On average, the faded-bar participants took longer to select an answer for 10 of the 15 questions. This could be due to a lack of understanding of the faded-bar graphics, thus taking a longer time for participants to realize what they were looking at. Specifically, four of the questions in task 3 took faded-bar participants longer to select an answer than the boxplot participants. The boxplot climate snapshot color ramp had larger values associated with darker colors and smaller values associated with lighter colors, which is appropriate for communicating data values (Schloss et al. 2019). The faded-bar version had dark colors representing both smaller and larger values, which could make it harder to interpret and understand (Netek et al. 2018). From the analysis, the boxplot version of CAnVAS was considered more efficient than the faded-bar version.
Effectiveness was measured by participants either correctly or incorrectly answering each question. For this study, the best way to represent effectiveness is by showing how many people answered the questions incorrectly. Overall, there were five questions for which the same number of people in both groups answered the questions incorrectly. There were six questions for which the faded-bar participants more frequently answered incorrectly than did the boxplot participants. There were four questions for which the boxplot participants more frequently answered incorrectly than did the faded-bar participants. Thus, the boxplot version of CAnVAS proved to be slightly more effective than the faded-bar version when looking at correct and incorrect answers. There could be many different factors that contributed to this result. For instance, the boxplot participants located the correct locations faster and fixated on them longer than faded-bar participants, which may have led to the boxplot participants answering more questions correctly.
Satisfaction was measured by thematic coding of the interviews conducted with undergraduates following their participation in eye tracking. Cocoding of interviews was completed for 10% of codes and received good agreement (Cohen’s kappa = 0.65). There were more times that participants identified parts of CAnVAS that were difficult to understand than easy to understand, which could be due to the lack of knowledge in this specific subject. Overall, participants thought the colors of the climate projections on CAnVAS were good for correlating between the legend and the data shown. Most participants were able to correctly interpret which color went with which dataset (i.e., red with average maximum temperature, blue with average precipitation, and orange with occurrence of maximum temperature). There were some recommendations provided during interviews directed toward the climate-projections portion of CAnVAS, suggesting that the majority of participants would have liked to see something different. Our suggestion to CAnVAS developers at SCONC was to split this section into separate graphs for each parameter instead of showing all parameter data on a single plot. Faded-bar participants also noted a preference to view plot data in a different format, even as a boxplot. This helped validate our suggestion to CAnVAS developers to use boxplots to visualize data on the graphs.
Also, codes were developed around task 3 to better understand whether participants actually looked at the climate snapshots or the climate projections of CAnVAS when answering the questions. Participants identified if they looked only at the climate snapshots (n = 7), only at the climate projections (n = 11), or both (n = 26). Some participants indicated that they only looked at the climate projections to answer questions on task 3, implying that they did not necessarily understand what they should have been looking for in this task, since questions were intended to have them use the climate snapshots. The majority of participants looked at both the climate snapshots and climate projections to answer the questions about task 3. Some participants identified they looked at both since they were already familiar with the climate projections and felt like it aided them to look at the climate projections to understand what exactly was being displayed on the climate snapshots. This result indicates that there could have been a more obvious explanation on CAnVAS of what exactly the climate snapshot was displaying, perhaps a visual to direct attention or more descriptive text embedded in the site. The boxplot version of CAnVAS was more satisfying to participants, which aligned with higher efficiency and effectiveness with this version.
Our results align with previous literature that has used eye tracking to understand how users engage with climate figures and/or DSS tools. For instance, the boxplot version of CAnVAS had a color ramp where smaller values are associated with lighter colors and larger values are associated with darker colors, which is seen as appropriate for communicating data values (Schloss et al. 2019). The suggestion of adding a description or visual aid to allow participants to better understand task 3’s purpose aligns with what was previously found in the literature, where suggestions have been made to emphasize what information is most important within DSSs (i.e., an arrow pointing toward certain information; Courtney 2019, p. 62; Maudlin et al. 2020). Representations of climate information both spatially (climate snapshots) and temporally (climate projections) were effective at communicating climate information as shown by participants looking at both sections to answer questions. This aligns with results in Davis et al. (2020) where it was found that data presented for a specific location was most effective when displayed temporally and spatially, even if it was not the end goal of this CAnVAS task to look at both representations. However, our method differs from previous literature in that this current study uses eye tracking to aid in the development and critique of DSS rather than ranking of criteria or other non-eye-tracking evaluation methods (Oztekin 2011; Arciniegas et al. 2013; Bagstad et al. 2013; Zhang et al. 2015; Han et al. 2018).
5. Conclusions
In this study, coproduction was employed to produce a pilot tool for planned use by USFWS biologists. Interactions with an initial USFWS committee to assess the needs of SSA authors led to the development of mock-ups of the CAnVAS interface, which were then tested using an A/B study design and a combination of eye tracking and interview approaches with novice users. This research was able to provide recommendations on how to improve the CAnVAS DSS specifically in regard to how quantitative information is displayed in the tool. The following recommendations refer to what aspects of CAnVAS could be retained or modified for improvement after this first iteration of the study, which will be referred to as CAnVAS 2.0.
CAnVAS 2.0 should keep the same colors for displaying the climate projections on the graph, since participants were able to verbally identify correct interpretations of the colors used. Previous research suggests that colors should be related to the displayed data, and it is evident that the CAnVAS climate projections meet this requirement (Netek et al. 2018). This new version should have the data displayed as boxplots because, for most of the tasks, it took faded-bar participants longer to fixate on the correct locations. Also, the boxplot version of CAnVAS was more efficient and effective than the faded-bar version. CAnVAS 2.0 should include more obvious ways to show that the data from the climate projections are being visualized in the climate snapshots by having an explanation that appears when a user hovers over the component with their mouse. The color ramp of the climate snapshots should be the same coloring as the boxplot version. Although these changes are recommended for CAnVAS, they could certainly resonate with others that may be creating DSSs for stakeholder use and provide guidance on how to display certain data and specific information, as this is often an early step to consider in DSS development.
The next version of CAnVAS will convert the still-image mock-ups into a live website where users can select endangered species and locations, and relevant climate information will be displayed. USFWS biologists will be able to use this tool to evaluate climate risks for regional endangered species across their native range or other key locations. After refinement of CAnVAS, another eye-tracking research study with the targeted stakeholders in the USFWS will be conducted. As after the first evaluation stage described in this study, CAnVAS will be refined again through this iterative development process informed by usability research, while also maintaining collaboration between the science makers and science users. Previous research highlights the importance of stakeholder engagement with DSS evaluation, and this research will not only allow for engagement but allow for end-user testing to improve this DSS (Wong-Parodi et al. 2020). Testing CAnVAS with the end user is important to the DSS development process to ensure the USFWS stakeholders are able to accurately interpret the scientific information provided and use it for everyday decision-making (Davis et al. 2020). These components will contribute to an improved DSS to ultimately benefit USFWS decision-making. For other researchers who would like to make similar DSS tools with a particular stakeholder, we suggest a coproduced user approach such as that used in the example provided in this research, which includes an evaluative process to test the usability of the coproduced product as it is being developed.
Acknowledgments.
This material is based in part upon work supported by the National Science Foundation under Grant NSF-DGE-1922687 and the U.S. Geological Survey’s Southeast Climate Adaption Science Center under Grant G00013406. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or the U.S. Geological Survey. Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. government. Extended thanks are given to Dr. Lindsay Maudlin, Dr. Steph Courtney, and Jena Brown for their help with cocoding and eye-tracking expertise. Also, thanks to the participants of this study and the reviewers of this paper.
Data availability statement.
Because of their proprietary nature, supporting data cannot be made openly available. Details of the data and how to request access are available from author Cashwell at Auburn University.
REFERENCES
Aldridge, H. D., C. Davis, and R. Boyles, 2016: PINEMAP decision support system. USDA, accessed 29 October 2019, https://climate.ncsu.edu/pinemap/index.php?tool=dev.
Arciniegas, G., R. Janssen, and P. Rietveld, 2013: Effectiveness of collaborative map-based decision support tools: Results of an experiment. Environ. Modell. Software, 39, 159–175, https://doi.org/10.1016/j.envsoft.2012.02.021.
Atkins, R. M., and K. S. McNeal, 2018: Exploring differences among student populations during climate graph reading tasks: An eye tracking study. J. Astron. Earth Sci. Educ., 5, 85–114, https://doi.org/10.19030/jaese.v5i2.10219.
Bagstad, K. J., D. J. Semmens, S. Waage, and R. Winthrop, 2013: A comparative assessment of decision-support tools for ecosystem services quantification and valuation. Ecosyst. Serv., 5, 27–39, https://doi.org/10.1016/j.ecoser.2013.07.004.
Carpenter, P. A., and P. Shah, 1998: A model of the perceptual and conceptual processes in graph comprehension. J. Exp. Psychol. Appl., 4, 75–100, https://doi.org/10.1037/1076-898X.4.2.75.
Corner, A., C. Shaw, and J. Clarke, 2018: Principles for effective communication and public engagement on climate change: A handbook for IPCC authors. Climate Outreach Doc., 28 pp., https://www.ipcc.ch/site/assets/uploads/2017/08/Climate-Outreach-IPCC-communications-handbook.pdf.
Courtney, S., 2019: Building better graphs for climate change communication: Evidence from eye-tracking. M.S. thesis, Dept. of Geosciences, Auburn University, 71 pp., https://etd.auburn.edu/bitstream/handle/10415/6628/SLCourtney%20Thesis.pdf?sequence=2.
Creager, J. H., and D. J. Gillan, 2016: Toward understanding the findability and discoverability of shading gradients in almost-flat design. Proc. Hum. Factors Ergon. Soc. Annu. Meet., 60, 339–343, https://doi.org/10.1177/1541931213601077.
Davis, C., H. Aldridge, R. Boyles, K. S. McNeal, L. Maudlin, and R. Atkins, 2020: Visually communicating future climate in a web environment. Wea. Climate Soc., 12, 877–896, https://doi.org/10.1175/WCAS-D-19-0152.1.
Durose, C., C. Mangan, C. Needham, and J. Rees, 2014: Evaluating co-production: Pragmatic approaches to building the evidence base. Political Studies Association Conf., Manchester, United Kingdom, PSA, https://www.psa.ac.uk/sites/default/files/conference/papers/2014/Evidencing%20Coproduction%20PSA%20paper,%20final.pdf.
Ehmke, C., and S. Wilson, 2007: Identifying web usability problems from eye-tracking data. Proc. 21st British HCI Group Annual Conf., Lancaster, United Kingdom, Electronic Workshops in Computing, https://doi.org/10.14236/ewic/HCI2007.12.
Haesner, M., J. Chandra, A. Steinert, and E. Steinhagen-Thiessen, 2018: An eye movement analysis of web usability: Differences between older adults with and without mild cognitive impairment. Assistive Technol., 30, 66–73, https://doi.org/10.1080/10400435.2016.1251994.
Haines, A., and K. Ebi, 2019: The imperative for climate action to protect health. N. Engl. J. Med., 380, 263–273, https://doi.org/10.1056/NEJMra1807873.
Han, T., Q. Huang, A. Zhang, and Q. Zhang, 2018: Simulation-based decision support tools in the early design stages of a green building—A review. Sustainability, 10, 3696, https://doi.org/10.3390/su10103696.
Hansen, J., M. Sato, R. Ruedy, K. Lo, D. W. Lea, and M. Medina-Elizade, 2006: Global temperature change. Proc. Natl. Acad. Sci. USA, 103, 14 288–14 293, https://doi.org/10.1073/pnas.0606291103.
Harold, J., I. Lorenzoni, T. F. Shipley, and K. R. Coventry, 2016: Cognitive and psychological science insights to improve climate change visualization. Nat. Climate Change, 6, 1080–1089, https://doi.org/10.1038/nclimate3162.
Hegarty, M., M. S. Canham, and S. I. Fabrikant, 2010: Thinking about the weather: How display salience and knowledge affect performance in a graphic inference task. J. Exp. Psychol. Learn. Mem. Cognit., 36, 37–53, https://doi.org/10.1037/a0017683.
King, A. J., N. Bol, R. G. Cummins, and K. K. John, 2019: Improving visual behavior research in communication science: An overview, review, and reporting recommendations for using eye-tracking methods. Commun. Methods Meas., 13, 149–177, https://doi.org/10.1080/19312458.2018.1558194.
Manson, S. M., L. Kne, K. R. Dyke, J. Shannon, and S. Eria, 2012: Using eye-tracking and mouse metrics to test usability of web mapping navigation. Cartogr. Geogr. Inf. Sci., 39, 48–60, https://doi.org/10.1559/1523040639148.
Marlon, J., L. Neyens, M. Jefferson, P. Howe, M. Mildenberger, and A. Leiserowitz, 2022: Yale climate opinion maps 2021. Yale Program on Climate Change Communication, accessed 25 July 2022, https://climatecommunication.yale.edu/visualizations-data/ycom-us.
Maudlin, L. C., K. S. McNeal, H. Dinon-Aldridge, C. Davis, R. Boyles, and R. M. Atkins, 2020: Website usability difference between males and females: An eye-tracking evaluation of a climate decision support system. Wea. Climate Soc., 12, 183–192, https://doi.org/10.1175/WCAS-D-18-0127.1.
McNie, E. C., 2013: Delivering climate services: Organizational strategies and approaches for producing useful climate-science information. Wea. Climate Soc., 5, 14–26, https://doi.org/10.1175/WCAS-D-11-00034.1.
Meadow, A. M., D. B. Ferguson, Z. Guido, A. Horangic, G. Owen, and T. Wall, 2015: Moving toward the deliberate coproduction of climate science knowledge. Wea. Climate Soc., 7, 179–191, https://doi.org/10.1175/WCAS-D-14-00050.1.
Moreland, K., 2009: Diverging color maps for scientific visualizations. Proc. Fifth Int. Symp. Visual Computing, Las Vegas, NV, ISVC, 92–103, https://doi.org/10.1007/978-3-642-10520-3_9.
Netek, R., T. Pour, and R. Slezakova, 2018: Implementation of heat maps in geographical information system—Exploratory study on traffic accident data. Open Geosci., 10, 367–384, https://doi.org/10.1515/geo-2018-0029.
Nisbet, M. C., 2009: Communicating climate change: Why frames matter for public engagement. Environ. Sci Policy Sustainable Dev., 51, 12–23, https://doi.org/10.3200/ENVT.51.2.12-23.
Okan, Y., M. Galesic, and R. Garcia-Retamero, 2016: How people with low and high graph literacy process health graphs: Evidence from eye-tracking. Behav. Decis. Making, 29, 271–294, https://doi.org/10.1002/bdm.1891.
Oztekin, A., 2011: A decision support system for usability evaluation of web-based information systems. Expert Syst. Appl., 38, 2110–2118, https://doi.org/10.1016/j.eswa.2010.07.151.
Pyke, C. R., B. G. Bierwagen, J. Furlow, J. Gamble, T. Johnson, S. Julius, and J. West, 2007: A decision inventory approach for improving decision support for climate change impact assessment and adaptation. Environ. Sci. Policy, 10, 610–621, https://doi.org/10.1016/j.envsci.2007.05.001.
Reidmiller, D. R., C. W. Avery, D. R. Easterling, K. E. Kunkel, K. L. M. Lewis, T. K. Maycock, and B. C. Stewart, Eds., 2018: Impacts, Risks, and Adaptation in the United States. Vol II, Fourth National Climate Assessment, U.S. Global Change Research Program, 1515 pp., https://doi.org/10.7930/NCA4.2018.
Schloss, K. B., C. C. Gramazio, A. T. Silverman, M. L. Parker, and A. S. Wang, 2019: Mapping color to meaning in colormap data visualizations. IEEE Trans. Visualization Comput. Graphics, 25, 810–819, https://doi.org/10.1109/TVCG.2018.2865147.
Shah, P., and E. G. Freedman, 2011: Bar and line graph comprehension: An interaction of top-down and bottom-up processes. Top. Cognit. Sci., 3, 560–578, https://doi.org/10.1111/j.1756-8765.2009.01066.x.
Somerville, R. C. J., and S. J. Hassol, 2011: Communicating the science of climate change. Phys. Today, 64, 48–53, https://doi.org/10.1063/PT.3.1296.
Turton, T. L., C. Ware, F. Samsel, and D. H. Rogers, 2017: A crowdsourced approach to colormap assessment. EuroVis Workshop on Reproducibility, Verification, and Validation in Visualization, K. Lawonn, N. Smit, and D. Cunningham, Eds., Eurographics Association, https://doi.org/10.2312/eurorv3.20171106.
VanderMolen, K., A. M. Meadow, A. Horangic, and T. U. Wall, 2020: Typologizing stakeholder information use to better understand the impacts of collaborative climate science. Environ. Manage., 65, 178–189, https://doi.org/10.1007/s00267-019-01237-9.
Wall, T. U., A. M. Meadow, and A. Horganic, 2017: Developing evaluation indicators to improve the process of coproducing usable climate science. Wea. Climate Soc., 9, 95–107, https://doi.org/10.1175/WCAS-D-16-0008.1
Wong-Parodi, G., K. J. Mach, K. Jagannathan, and K. D. Sjostrom, 2020: Insights for developing effective decision support tools for environmental sustainability. Curr. Opin. Environ. Sustainability, 42, 52–59, https://doi.org/10.1016/j.cosust.2020.01.005.
Zhang, D., X. Chew, and H. Yao, 2015: Development of a prototype web-based decision support system for watershed management. Water, 7, 780–793, https://doi.org/10.3390/w7020780.
