a. Trust in original historical weather and climate content

Most researchers working in the field of historical data rescue rely on post-transcription statistical methods to assess the quality of data, sometimes guided by metadata such as station history for potential inhomogeneities (e.g., weather station relocations, observer details, or instrument changes) (e.g., Auer et al. 2007; Chandler et al. 2012). Where there is pretranscription guidance, Brönnimann et al. (2006) suggest filling temporal, spatial, or other knowledge gaps, inferring fitness for intended modern usage and augmenting datasets with metadata. In Canada, long-term continuity is important for building a historical precipitation dataset (Mekis and Vincent 2011); for temperature, factors include “data quality, spatial and temporal coverage, and longevity” (Vincent et al. 2012, p. 2). For the U.S. Climate Data Modernization Program, “the proper usage of the newly digitized data depends upon an assessment of the history and the quality of the observations” (Dupigny-Giroux et al. 2007, p. 1017). Trustworthiness is implicit when quality is mentioned, although temporal and spatial coverage are also mentioned.

Assessments of historical context, whether recorded in the past or offered in contemporaneous accounts, can offer insight into observations. Some studies emphasize familiarity, on the part of both original observers and rescuers of those observations, with the weather of a place as a way to impute observation quality (e.g., Nash and Endfield 2002). Researchers have built confidence in historical observations by detailing the historical training and social setting of the observer, such as Hestmark and Nordli (2016), who discuss the background of observer J. Esmark in early nineteenth-century Oslo. Camuffo (2002) describes a historical tradition of the meteorological observations in Padova, Italy, as far back as Galileo. Demarée and Ogilvie (2008, p. 424) report on the value of observations from Moravian missionaries “who combined their evangelizing activities with a keen interest in the natural sciences.” Qualitative details about observers’ backgrounds and affiliations imply individuals’ skill, motivation to do good, and consistency in ensuring data quality.

b. Trust in citizen science and crowdsourced efforts

Trust is a crucial research domain in citizen science because credentialed scientists increasingly rely on amateurs. Certain research like large ornithological studies depend on data from eBird (https://ebird.org) or the Audubon Society’s long-running Christmas bird count (Cooper et al. 2014). Weather transcription efforts often rely on volunteers because transcription conducted by professionals or by paid data entry employees can be cost prohibitive. These nonexperts can be more than stenographers; they can identify discrepancies, problems, and occasionally correct original observations (Lewandowski and Specht 2015). Using citizen science in weather, agencies like the Australian Bureau of Meteorology can “incorporate . . . local knowledge and historical accounts” (Pecl et al. 2015, p. 3).

Reliance on nonexpert transcription drives research on the accuracy of data produced by citizen scientists. After the fact, Cohn (2008) find that seventh grade students had an accuracy rate of 95% in a citizen science project identifying biological specimens. Ryan et al. (2018) compare results of student transcription as part of a course to transcription by Met Éireann (the Meteorological Agency of Ireland) personnel and find a cumulative error rate of less than 1%. Historical weather data offer few benchmarks from professionally produced data with which to compare accuracy in citizen science transcribed data (e.g., Dupigny-Giroux et al. 2007; Lewandowski and Specht 2015). Hunter et al. (2013) develop a set of metrics to assess citizen science that could be useful for weather data rescue projects, including a contributor’s role and qualifications (e.g., primary/secondary/Ph.D. student, volunteer, council worker, scientist) and a quantitative measure of valuation (e.g., frequency and amount of contributions, level of training) (cf. Mateus et al. 2021). Whether indicators of skill or capacity, these inferential methods can assess quality of contributions.

Quality of contributions in citizen science research can be improved if we attend to citizen scientists’ motivations to contribute (e.g., Gilfedder et al. 2019). Primary reported motivations include participating in original scientific inquiry, learning about new fields, gaining recognition for their contributions, and pursuing a hobby (Raddick et al. 2009; Rotman et al. 2014; Land-Zandstra et al. 2021). Historical weather citizen scientists may be driven by additional motivations such as interests in history, genealogy, or geography (Sieber and Slonosky 2019). Research into the citizen science platform Old Weather (https://www.oldweather.org) finds that individuals contribute when they experience a sense of “connectedness and membership” (Oomen and Aroyo 2011). Levels of contribution to Old Weather also can vary by category of motivation: Intrinsic (e.g., curiosity, contributions to science) and extrinsic motivations (e.g., leaderboards) (Eveleigh et al. 2014). Superusers report intrinsic motivations; occasional transcribers are attuned to extrinsic features, like specific tasks. We align motivation to do good with benevolence in trust.

Eveleigh et al. (2014) find that the structure of the transcription interface can attract even the “dabblers,” who will be more motivated to transcribe when each transcription activity is relatively small or when the site is designed to assuage anxiety over transcription quality. Mugar et al. (2015) and Skarlatidou et al. (2019) show the role played by user interface (UI) design in attracting citizen scientists. They demonstrate that numerous design elements improve trust in contributions. Such elements include descriptions of elements on the homepage, prominent positioning of assistance features such as tutorials and frequently asked questions (FAQs), ability to participate in forums or to annotate contributions, use of images or pulldown menus, ability for the contributor to see the data soon after it has been collected, use of scientific and lay language together, and consistency in naming elements in the UI. By enhancing contributors’ user experience, attentive platform design can improve trust in the consistency of transcriptions.

c. Our organizing trust framework

We argue that historical weather rescue can be conceptualized as a life cycle that moves from trust in original observers and their observations onto trust in current transcribers and current transcriptions. We operationalize trust as skill, benevolence, and consistency. Trust can be accumulated at several steps, which is important because researchers may possess insufficient information in any one step to determine whether the trust level is sufficient to prioritize data rescue. Figure 1 shows our framework, where step-by-step we establish trustworthiness. In that way, trust is both a process and an outcome.

View Full Size

Our organizing framework, which links trust as a function of skill, benevolence, and consistency in four steps: from trust in historical weather observers and observations to trust in the present among transcribers and transcription methods of those observations. Steps can proceed in a linear manner, but one step can exert influence on a prior step, e.g., when a present-day transcriber identifies an error in a past observation or finds information in the text about a particular observer.

Citation: Weather, Climate, and Society 14, 3; 10.1175/WCAS-D-21-0077.1

• Download Figure
• Download figure as PowerPoint slide

It is important to acknowledge the limits of trust. We describe a framework on which to make decisions pretranscription. Once the observations are transcribed, they enter a new phase where more objective, quantitative, posttranscription validation can be applied (cf. Bonter and Cooper 2012; Brinkerhoff et al. 2017; Hunter et al. 2018). At the outset, there is no way of verifying an observation of, say, 31°C from a transcribed weather diary from a day in 1880 at a location in Australia as true. Instead, we infer that the best explanation for the database entry is that the temperature was consistently recorded and that the record was accurately kept and transcribed. We rely on trust rooted in our judgment and a qualitative assessment of data reliability.

a. Why do we trust the observers?

The question of whether observers in meteorology need to be formally accredited in the field to produce valuable data has received some attention (e.g., Endfield and Morris 2012). Our case studies show many, though not all, observers were not credentialed as meteorologists but had professional training in other fields. Observers for Weather Detective belonged either to the Royal Navy or merchant marine. Some ACRE-Canada observers were associated with the military (Royal Engineers or Royal Artillery); others were educators, doctors or associated with the law. DRAW was associated with the field of higher education at McGill University. Both SEARCH and NEHHWD include diaries from graziers who observed the weather on their farms for many decades so they could deepen their knowledge of the land. In Australia, convicts were trained to keep records (Ashcroft 2014). In Canada, volunteers from a wide variety of occupations, including religious men and women and lighthouse keepers, contributed to the public Meteorological Service of Canada (Kingston 1875).

Nonprofessional does not mean nonexpert, especially in the eighteenth and early nineteenth century when much scientific activity, particularly in the Anglosphere, was “amateur” (Silvertown 2009). Education in law, medicine and engineering emphasized careful and precise observation and diligent record keeping. Ashcroft et al. (2014, p. 160) find that “[m]any of the observers [in southeastern Australia pre-1860] were scientific men,” suggesting that they were connected with the scientific community, familiar with meteorological theories of the time, and conversant with principles of instrument exposure and calibration. Some observers whose professional duties involved maintaining meteorological records continued observing the weather in a private capacity once they retired (e.g., F. Abbott in Tasmania, Australian surveyor P. Parker King, and Canadian Artillery Bombardier T. Menzies; Moyal 2003; Ashcroft et al. 2014; Slonosky 2018; Rimmer 2020). Professional background, training in weather observing, or scientific interests suggest skill in our characterization of trust.

The SEARCH project places a high value on understanding the motivation of individual observers, whether individuals observed out of personal interest, professional duty or were paid because this informs trust in observers’ willingness to collect consistent results (Ashcroft et al. 2014). The biographies of many ACRE-Canada observers are documented in Slonosky (2018) and show a mix of motives, from documenting climate change to understanding disease. Some observers were keen to ensure their records captured unusual weather events. In ACRE-Canada, we note the enthusiasm and dedication in supernumerary recordings outside the usual or prescribed observation period, such as overnight precipitation events and extra observations during storms recorded by McCord (Fig. 2c). McCord had no formal scientific background but acquired sufficient knowledge during his lifetime to be named to numerous scientific societies (Slonosky 2018). In SEARCH, the observer at Sydney’s South Head station recorded rainfall observations approximately every hour during several extreme rain events in Sydney in the 1840s, as opposed to once a day as usual (Ashcroft et al. 2014). Observers’ dedication, their benevolence, implies that reports of high rainfall amounts during these events are trustworthy (Ashcroft et al. 2019).

Dedication also can be seen in observers’ reviews of their own observations. DRAW records contain visible corrections of observations. Observatory director and principal observer C. McLeod (McLeod 1879) wrote on 25 March 1879 that “for about a month back the 4:48 observation has not been taken regularly. The observer having been in the habit of manufacturing it from the 1:48 obs’n.” The observations at 0448 local time were struck through but not erased. Observers often left notes, comments, letters, diaries, additional calculations, and printed articles concerning their observations. They include comments on instruments such as maker, calibration, exposure, and comparisons with other observers’ instruments or observations, letters to peers and instrument makers, and printed articles in contemporaneous journals describing instruments and observatories (e.g., McCord 1843; Hall 1858; Bridgman et al. 2019).

We find instances in which the social network of the observer, such as how much one observer contacted other observers or what networks and social support were available, indicated the quality of the observations. This is especially so with observers who had no scientific background and learned through their participation in networks and correspondence (e.g., Slonosky 2018). For example, the advice of influential scientist J. Herschel to weather observers was widely distributed among amateur observers in the English-speaking world. His advice spread to T. Lempriere in Australia (SEARCH) and McCord (Herschel 1840) in Canada (ACRE-Canada). McCord, Smallwood, and Hall were part of local societies such as the Natural History Society of Montreal, where they presented their observations and results, as well as North American networks such as the Smithsonian volunteer weather observers’ network.

As the networks and communications grew more sophisticated, coordinating agencies sent out representatives and instructions to standardize observations. In SEARCH Australian Government Astronomers H.C. Russell, C. Todd, and C. Wragge developed training for observers in telegraph-connected locations (Home and Livingston 1994), sending calibrated instruments by train or by horse and carriage, writing thousands of letters and providing structured observation books to ensure standardized observations from volunteer observers. Kingston did the same in Canada, sending instruments by ship (e.g., Australian Museum of Applied Arts and Sciences 2008; Slonosky 2018). We find in ACRE-Canada that the Smithsonian provided preprinted forms with instructions for observations across North America. Following advice and standards provided by these networks suggest a degree of trust in consistency among observers.

b. How do we decide we can trust the original observations?

Here we look for observations that are skillfully, benevolently, and consistently recorded. To establish trust in the observations prior to transcription, we begin with the sources.

With SEARCH, we sought published observations from government collections rather than newspaper reports or casual correspondence as we were able to find more detailed metadata for these observations than, for example, values published in the newspaper.

For ACRE-Canada (Fig. 2c) and DRAW (Fig. 2d) we look to observations originating from collections in archival sources, such as the McGill University Archives, the McCord Museum Archives, and Canadian national and provincial archives. Weather Detective (Fig. 2b) used ships’ logs for the data source, which were mandatory records of the ship movements (Wheeler 2014). SEARCH and Weather Detectives assume that a report or mandated compilation were subject to a certain level of vetting to ascertain data quality; observations found in personal diaries and other casual documents, such as is more often the case in ACRE-Canada, will not necessarily have been checked. Additionally, a high volume of potential data in a special collection such as DRAW is easier to compile and assess than occasional weather observations found in other materials like newspaper articles or diaries. Large volumes of regularly formatted materials can be easier to transcribe. Throughout, we must be mindful of privileging official or formatted accounts in projects such as Weather Detectives or DRAW over original handwritten diaries in projects such as ACRE-Canada. Formalized observations can contain their own complexities (e.g., modifications by observers) but other sources like private weather diaries records can be more difficult and time intensive to find.

We make informed trade-offs in our choices, for instance, whether to transcribe all the records of one early instrumental network versus many weather diaries over larger geographical areas. One way to make an informed judgment is to check sources, although source checking does not confer trust upon all items within a given collection (e.g., an individual ledger book with hourly records, sunshine records, and wind records). We choose items in which observations were made at regularly observing times; were made daily or preferably several times a day; included daily or subdaily instrumental measurements such as thermometer or barometer; had continuous observations on all or almost all days without regularly missing days; and showed values consistent with expected season, location, and climate from a visual inspection of the manuscript data. An initial perusal of register books from the period 1874–1930 for DRAW found observations to be acceptably complete, precise, and legible, although subsequent examination before and after transcription has found observing times to be more irregular than first realized. This pre- to postexamination highlights the limits of trust, in terms of consistency. The occasional addition of observations (e.g., during a storm) suggests benevolence but comes at the expense of consistency.

Certain records contain multiple attributes, which allow us to preliminarily trust the observations based on the number and consistency among variables. DRAW logbooks contain temperature, barometer, precipitation, wind direction; some logbooks contain a catchall section called “Weather” or “Remarks” (“rain,” “clear”). We can visually inspect a sample for coherence across a subdaily record. If it was snowing then was the temperature below freezing? If a thunderstorm was recorded, was it warm enough for convection? In SEARCH, we explore whether temperatures were as high as expected during summer drought events (Ashcroft et al. 2014). Visual inspection of observations prior to transcription also can help us identify potential quality issues, such as multiple days with identically repeated values or a large number of observations rounded to the nearest 0 or 5, rather than more evenly spread numbers. Sometimes variability within an attribute is key. Figure 2c displays an ACRE-Canada page with variations in temperature from cold to thaw that suggests expected seasonality and demonstrates the skillful aspect of trust. We note that familiarity with the climate and local conditions is helpful for the researcher in deciding to trust the data. Section 4c suggests that local or acquired knowledge from the transcriber community may assist the researcher as well.

Important trust criteria for historical observations include known location and continuity over time. Not all data sources satisfy these criteria; for certain places, the only observations available are less than ideal. It is rare to find all these criteria in one source document. In SEARCH, short sets of observations for four different stations on Australia’s east coast for 1821–22 are the only observations for the country in that time period. Occasionally having multiple stations observing at the same time can increase trust because these observations were likely part of a campaign and incorporated observer training and coordinated calibration.

Knowledge of instruments affirms trust, via skill and consistency, in observations. Nineteenth-century meteorological instruments were often large, cumbersome, and expensive, and prone to breakage and calibration issues (referenced in Fig. 2c). Calibration was no trivial task, as the instrument needed to be transported to and from an official observatory (e.g., by ship, rail, canoe, or horse and carriage) and compared with the official instruments. Many early observations in Australia and Canada are marred by gaps due to uncalibrated instruments or instruments that broke enroute. When exploring the diaries for NEHHWD, for example, we noticed a missive from Russell, who wrote to a regional observer in New South Wales in 1879, “it is quite possible that the barometer . . . is in error because, it assumes that [it] is correct after a long, and in olden time very rough journey, which may put it out of order” (Thornton et al. 2018).

Accurately measuring extremely low temperatures has proved difficult in Canada for centuries. Certain instrument types may produce higher uncertainties for values below a certain threshold and subsequently lower the trust in those observations. Problems with instruments often arose because of questions observers had about their instruments. We found that standardized instruments were purchased for some observers by central organizing offices such as the Meteorological Service of Canada (Kingston 1880) or available for purchase via the Smithsonian network (Smallwood 1858).

Last, we increase our trust when observations contain “show your work” calculations within the body of the record. DRAW ledgers contain several columns of raw observations and derived measurements, such as barometer adjusted for thermal expansion of mercury. Pages also contain intermediate calculations such as sums and averages. By having original plus intermediate values recorded, any errors can be identified and corrected, further increasing trust in skill and consistency.

c. Why do we trust the transcribers?

We rely on similar measures of trust in the transcribers to record the original observations skillfully, benevolently, and consistently. We find many transcribers to be highly engaged and dedicated, considered a contribution to benevolent trust, in the transcription process. For instance, ACRE-Canada transcribers added image file names to the transcription spreadsheets for traceability, transcribed summary and mean calculations, and compared these historically calculated values with mean values calculated from transcribed data in spreadsheets. One contributor found a “ruler” app to ensure correct data entry.

Site analytics can reveal transcriber interactions with the site. Transcribers accessed the tutorials page on the DRAW website over 1100 times between 1 January 2019 and 23 April 2021; the FAQ pages were accessed 494 times. The interactions suggest users were concerned about making mistakes. Bellman et al. (2016) conducted focus groups to assess the DRAW user experience. They find that citizen scientists were so anxious about making mistakes they required reassurance that their contributions would be validated by others. If transcribers exhibit such concern about the quality of their contributions and attempt to gain skills then we can begin to trust the accuracy of their contributions.

We also can base trust in the transcriber on rating or ranking metrics, where we consider our transcribers to be contributors as used in the crowdsourcing literature. Using contributor metrics as a replacement or augmentation for contribution accuracy has become acceptable practice for crowdsourced big data (Goodchild and Li 2012). Contributor metrics can be operationalized, for example, as the number of contributions completed by an individual or the length of time contributors have participated. Figure 3 provides the contribution patterns in DRAW. This heavy skew is common in crowdsourcing and citizen science in which a very few superusers greatly exceed the contributions of others (Gura 2013; Brovelli et al. 2020). With experience, some transcribers become as much, or sometimes more expert in individual observers or diaries than the project leads. DRAW’s superusers often notice anomalies or inconsistencies in the records and alert project organizers to data that require further inspection. Superusers can become highly skilled transcribers, which offers yet another reason why we trust transcribers, at least certain users.

View Full Size

Histogram of number of contributions from each DRAW transcriber, as of 31 Mar 2021.

Citation: Weather, Climate, and Society 14, 3; 10.1175/WCAS-D-21-0077.1

• Download Figure
• Download figure as PowerPoint slide

As with original historical observers, “nonprofessional” does not necessarily mean nonexpert. We do not know the backgrounds of all contributors—citizen science platforms often allow anonymity—but we do have information for some. Several participants in ACRE-Canada and DRAW possess credentials like those seen among historical observers: retired meteorologists and technicians, including those from the military or government, archivists, historians, computer specialists, and teachers. In DRAW, a sizable contingent consists of students or recent graduates with training in meteorology, environmental science, history, archives, and information studies (Bellman et al. 2016). Most are volunteers, although Weather Detective and SEARCH compensated some students to enter data.

Transcribers express several motivations for our data rescue projects, for example, to improve understanding of their local weather and climate and contribute to a global database of information that could “make a difference” in understanding climate change. We also see a strong historical element of motivation: Participants may wish to learn more about their own genealogy or the history of their community or are drawn to a time period covered by the weather observations. In Weather Detective, transcribers found records from ships on which their ancestors sailed. Many volunteers say that they are motivated by altruism. Some students report being motivated to add experiences to their resumés.

As with original observations, we see evidence of social networking in our projects. Volunteers who contributed to NEHHWD were found through personal connections as well as an online call for volunteers. This added to an inherent trust that came from the coordinators knowing some volunteers in person and or through lengthy email conversations. Additionally, NEHHWD relied on tiered content moderation approach, with retired individuals, a meteorologist and a historian, who coordinated contributions. Contributors to ACRE-Canada started a photo sharing site to exchange screenshots of difficult alphabetic characters to see if others could decipher the entries. Trust therefore can be characterized as consistency when transcribers adhere to norms created by the network.

d. Why do we trust the transcriptions?

Transcription of weather-related information from one format to another is not new. Herschel (1835) provided best practices for transcribing observations from one logbook format to a format that could be easily shared. As designers of weather citizen science projects, we realize that we too have a role in ensuring trust of the transcriptions. Because our cases rely heavily on citizen science, trusting transcriptions means developing robust platforms for volunteer engagement.

Predesigned templates play a crucial, if largely hidden, role in facilitating consistent transfer of information from historical media to machine readable formats (World Meteorological Organization 2016). Detailed spreadsheet templates aided in the transcription of data for NEHHWD (Bridgman et al. 2019) and project leads redesigned the templates into more convenient formats in response to contributors’ feedback. The ACRE-Canada project lead designed spreadsheet templates for each different diary type.

Both the Weather Detective and the DRAW leads developed web applications carefully tailored for the source material as a further way to ensure consistency. Figure 2d shows two pages from DRAW that are densely packed with numbers, meteorological symbols, and standardized abbreviations written in cursive handwriting. DRAW developers have customized the UI with drop-down select menus for fields, such as weather symbols, which could not be obviously transcribed by typing. To minimize uncertainty on the part of transcribers, drop-down select menus also are introduced for wind directions or cloud types, which limited entry options to abbreviations that matched the original (Sieber and Slonosky 2019).

All cases offer help pages or sections. Smaller projects, like NEHHWD or ACRE-Canada, allowed site administrators to send detailed instructions via individual emails. ACRE-Canada provided FAQs (readme files) in a DropBox folder containing image files and spreadsheet templates. Weather Detective and SEARCH had FAQs and help sections on their websites. DRAW has a help section on the website and explainers attached to each data entry element as well as video tutorials (https://vimeo.com/525062600). The Weather Detective site offered a tutorial showing how to read the observations and digitize the data (https://www.youtube.com/watch?v=qdYLxmEEe_g). Assistance features act to increase communications and provide training, which leads to an increase in trust in the area of skill. Improved skills in turn help to decrease errors, which increase consistency.

All projects create methods for volunteers to provide feedback on their transcriptions. Email exchanges in NEHHWD and SEARCH enabled transcribers to report if methodological errors were identified. ACRE-Canada had a communications forum and email so transcribers could confer with project investigators. DRAW has a feedback form as well as a dedicated email feedback account. Weather Detective had email and phone support for individual questions, for example, if volunteers could not read the handwriting or the text was in French. Two-way communication increases transcribers’ trust in us and our trust in them because we have evidence they were motivated and engaged.

Developers provide semiregular updates to contributors about project goals for SEARCH, Weather Detective, and DRAW. Volunteers have shared interesting case studies or publications they had found, via forums or directly with the project team. DRAW maintains a blog, a news page on the website, and a stand-alone newsletter. If project leads inform volunteers of the purpose of the data, especially if accuracy is important to that purpose, then research suggests we can better trust the data (Skarlatidou et al. 2019). It also suggests a “currentness” to the project; activities are ongoing and project leads care about informing participants (benevolence). Increasing communication and training increases involvement or benevolence, which increases skill and consistency, mutually reinforcing all aspects of trust.

This is not a paper on validation, which is a posttranscription activity; however, there are possibilities for some data validation during transcription. Preliminary validation on DRAW data by Brinkerhoff et al. (2017) find that 96.14% of the data was transcribed accurately for fields where accuracy was testable; 99.97% of the data transcribed fell within meteorological norms. Brinkerhoff et al. (2017) include a taxonomy of errors and find that misplaced decimal points, transposed digits, and mistaken punctuation marks were common sources of inaccuracies. Error rates from the ACRE-Canada project range from 0.02% to 0.20%. Validation of NEHHWD identifies less than 1% (0.80%) of the transcribed data required correction (Bridgman et al. 2019). There also are possibilities for data verification before or during transcription. An enhancement to the DRAW site enables self-verification: Users can review their own entries and add comments as deemed necessary (Sieber and Slonosky 2019). Weather Detective used a crowdsourcing approach, designing the system so each record was transcribed at least 3 times but often as many as 10 times. If the values were not identical then the value appearing most times was accepted.

a. Checking the checklist

The value of this checklist was affirmed after transcribing and validating case study data (cf. Slonosky 2014; Ashcroft 2014; Brinkerhoff et al. 2017; Sieber and Slonosky 2019). To further assess the utility of the checklist we apply it to a new case, a recently published dataset for Perth, Australia (Gergis et al. 2021). As we move through the steps, we can build a logical picture of reliability of observations before transcription is complete.

Step 1 exposes potential trust issues in the observations, as little is known about the observer of these weather records (Gergis et al. 2021). Notes within the weather diaries suggests that observers were scientifically trained as they were able to name cloud types and discussed calibration of various weather instruments.

The value and detail revealed when working through step 2 of the checklist can compensate for the lack of observer information. We can deduce that the source is relatively official as the 16 handwritten weather diaries transcribed as a part of the dataset came from a survey office. They are recorded on preprinted forms; they are regular and span a long period (1830–75). The observations also can be checked for internal consistency with qualitative and quantitative data available for cross referencing. Some information about instrumentation is provided. The observations are valuable because they fill a spatial and temporal gap, providing some of the earliest nineteenth-century weather observations we have for western Australia.

Whereas the transcriber elements of this project (step 3) did not fit in the standard citizen science mold; the checklist can still be applied to assess the trust we can put in their efforts. Two undergraduate students completing majors in atmospheric science worked together on a paid and then voluntary basis to transcribe the dataset. They were motivated by finances and academic opportunity as well their intrinsic interest as illustrated by the willingness of one student to continue in a volunteer role. The students also worked together and communicated regularly with other members of the team, giving them a strong social network. Both transcribers are authors on the study on which their efforts are based, as further recognition and high rating of their work.

The transcription elements used by Gergis et al. (step 4) reveal a small-scale setup and implementation, with templates designed to match the original observations. This included checking dates meticulously in the original diaries, as many Sundays were missing, and using conditional formatting to automatically highlight values that were outside expected thresholds. Instructions were provided to transcribers, with a direct line to the researchers involved in the project for feedback and questions. Multiple key entry was conducted at the start of the project (Gergis et al. 2021), with lessons learned from that experience fed back to the transcribers and used to improve the transcription process.

Overall, the checklist enables us to put moderate to high trust in the record described by Gergis et al. (2021) before conducting any statistical analysis of the dataset. Applying the checklist was quick and pragmatic and gave us an opportunity to consider the project from many angles before proceeding.

b. Cross connections and conclusions

Beyond applying individual elements, trust in data rescue represents an iterative, interlocking process of feedback loops. Many of our decisions about trust in observations lead back to the observer. Consistency in the observations suggests the observer exhibited benevolence and gained skills by observing all aspects of weather rather than just noticing extreme events or spectacular weather. This increases our trust in their observations. As suggested above, the source of the records can lead back to the observer. If the source is a government compilation (e.g., Australia) or specialized society source (e.g., The Natural History Society of Montreal) then it suggests not only that the observations were reviewed by peers but that the observer was sufficiently dedicated (i.e., benevolently motivated) to seek out credentialing and training. If present-day transcribers improve their knowledge of the observers and norms and practices surrounding observations being transcribed, then transcribers can better interpret the observations and be consistent in the observations they transcribe. Knowledge that an observer was originally from Scotland, for example, allowed one transcriber to discover that the word blinks was a Scots word for drizzle and allowed her to better interpret the handwriting of the diary. Examples also show how interactions between the transcribers and the platforms lead to an increase in trust in both the transcribers and the transcriptions through increased training leading to an increase in skill and consistency, while increased communication leads to an increase in benevolence.

Elements of the checklist are not mutually exclusive. For example, an original diary with a known observer and personal biographical information can use elements from steps 1 and 2. Information from sources such as printed national or international collections such as annual reports can fall under step 2. Should the content already be transcribed then we can assess the effectiveness of steps 3 and 4. We also may wish to revisit 1 and/or 2 to check if the original content is trustworthy.

Not all elements in Table 2 are necessary to confer trust with any given project. Insufficient information may exist to satisfy elements in the observation step. Conversely, sufficient information may exist to satisfy elements in the observer step, which then increases confidence in the resulting dataset. Not even our cases satisfy all the elements of the checklist, nor are the elements meant to replace other expert knowledge, which, for example, are needed to gauge expected seasonal values for the region. For nearly every case, we can think of a counterexample: A credentialed observer with links to networks whose observations proved less reliable when validated. Obscure, unknown diaries with no known social or professional links have produced highly accurate keystone records. Information is likely incomplete for any historical record. At minimum, this calls for careful documentation of the transcription process.

Elements of the list can be geographically, temporally, and also culturally contingent. Geographically and temporally, the field of meteorology evolved in different places at different rates from largely amateur in certain societies to more professionally organized in others. Historical weather data cover four centuries, six continents, all the oceans, and the upper reaches of the atmosphere, measured by dozens if not hundreds of instruments with continually evolving precision levels. Something as simple as the expected number of decimal places will thus vary by historical period and instrument. Expected seasonal variability changes with location, variable, and time step (hourly, daily, or monthly). Judgment needs to be exercised in conjunction with what information is available when evaluating historical observations. Traditions and organizations change between political and cultural milieus (e.g., military vs civilian, government sponsored vs amateur, independent or university organized). Historical data rescue is more than empirical validation. Trust is built on people and their cultures (e.g., Mayer et al. 1995; Rousseau et al. 1998).

In conclusion, we contend that trust in historical weather data rescue encompasses the entire life cycle of acquiring and transcribing observations. This includes trust in observers that considers observer skills and their benevolent motivations and consistency as well as trust in the observations, with issues such as consistency of source and instrumentation. In several cases, transcription has been undertaken by volunteer crowdsourcing or citizen science and therefore is connected to the quality of the transcription environment and the knowledge and passion of the transcribers. We hope that this checklist derived from our experiences assists in prioritizing the millions of historical weather observations waiting to be transcribed.