a. The data stream

When ARM started, there was hope that a single unified information system or database could be used to handle all of the data collected by the ARM Program. However, it was quickly realized that the variety of the instruments and sites resulted in a very diverse set of datasets. This diversity made it extremely difficult to store all of the data in a single database or to use a single unified information system. Thus, the program needed another approach.

Ultimately, the basic structure of ARM data became data streams of files, where a data stream was a set of files that came from a single source (e.g., an instrument or algorithm) with a fixed file structure. The data stream became a fundamental aspect of the ARM data structure. Nearly all data stream files contained measurements spanning a single day from a single instrument and location (e.g., from a surface meteorological station at the NSA site).

b. Adoption of NetCDF

ARM, like most scientific programs, understood the need to have metadata and documentation reliably associated with its data. For example, it is important to document the SI unit and minimum and maximum values for each geophysical variable in the data stream. As a consequence, ARM adopted the NetCDF file structure because it was a self-describing, machine-independent, and efficient binary data format with an extensive set of open-source software libraries for data access and file manipulation (Rew and Davis 1990; http://www.unidata.ucar.edu/software/netcdf/). A self-describing data format was very important because ARM data files would be moved and restructured in many ways during research use. This format allowed for global metadata about the entire data file (the date of creation, source of the information, programmatic citations, etc.) and metadata for specific data fields (descriptive strings about each field, valid minimums and maximums, precision, etc.). Machine independence was important because the research community used a variety of hardware (UNIX machines, PCs running windows, etc.) and software. Furthermore, binary data formats are more efficient from a storage perspective (i.e., the binary files are smaller than ASCII files).

ARM data were intended to be widely available, and open-source software, such as the NetCDF package, insured the ability to maintain access to the data without proprietary restrictions and prevented the software from becoming abandoned because of shifting corporate interests. The NetCDF library included routines that enabled users to directly access data fields by name with their own software without knowing detailed information about file layout and field position. Tools that came with the NetCDF software distribution readily enabled users to concatenate files across time for a single data stream and retain only a portion of the data fields (to reduce the number of data files and the data volume).

The decision to select NetCDF was easy because the data format was being developed specifically to support atmospheric research. NetCDF readily supported the time, spectral, and height dimensions needed for ARM data, thereby supporting the wide variety of data structures that were stored in NetCDF format. Data streams range from simple constructs such as simple time series of radiometric data to complex multidimensional arrays from cloud radars.

The selection of NetCDF had many of the positive features listed above. However, this decision also had negative impacts on the ARM user community. NetCDF libraries required the user to write software in a programming language (e.g., C, C++, and FORTRAN) to access and process the data, which was a significant challenge in the early days of ARM because NetCDF was a newly developed package. Not many in the atmospheric scientific community were familiar with it. Higher-level languages like Perl, Java, R, Python, MATLAB, and IDL with routines to access/read/write NetCDF files became more available in the mid-to-late 1990s, making it easier for scientists to use ARM data.

Each of these data access and processing methods required an investment in training, software development or software purchase to use the data. DOE management was initially concerned that the selection of NetCDF might restrict the use of ARM data by requiring the scientists to have programming skills to access the data, thereby limiting data use by scientists who could only use interactive tools like spreadsheets or text editors to “look at” data or create simple data plots. Thus, in the early 1990s, an experiment support team, which was a small group of data system staff, would work with ARM scientists to help them learn to manipulate ARM data in NetCDF files. This group was later called the ARM Experiment Center and also gathered requirements for commonly needed advanced data products. These efforts were the origin of ARM’s value-added products (VAPs; Ackerman et al. 2016, chapter 3). As newer tools became available and more scientists learned how to use NetCDF, this group was no longer needed to support ARM scientists, but efforts to design and create VAPs continued.

In retrospect, the decision to organize the data into daily NetCDF files and data streams as the basic elements for the ARM data structure has been a very robust and durable decision.

c. Raw versus processed data

There was an axiom that was popular in the early days of the ARM Program that is typically attributed to Gerry Stokes, the first chief scientist of the ARM Program: “one can survive bad analysis but cannot survive bad data” (G. Stokes 1996, personal communication). This idea resulted in the decision to save the raw output from each instrument in the ARM Data Archive as well as the more highly processed (i.e., calibrated, value added) data. While this approach consumed much more space in the ARM Data Archive, it also allowed problems in both simple ingest codes (i.e., processing codes that converted raw data to NetCDF) and VAP codes to be fixed. While this might seem obvious now, it was less so when the price of storage was markedly higher and the rate of data storage much slower than today.

d. Real-time to near-real-time availability

The goal of autonomous 24/7/365 operations required new thinking for making the data available to the scientific community; as Stokes (2016, chapter 2) indicated, there would be no respite at the end of the campaign for which to process/calibrate/quality control (QC) the data. Thus, the program’s original intent was to make the data available in real time. However, because of the relative infancy of the Internet and processing systems, data collection and processing took a finite amount of time. For a site that was well connected to the Internet such as the SGP, data could be made available to the scientific community within 2 days; however, for remote sites such as the TWP, the data would not be available for potentially several months because it was shipped back from the remote sites via transportable media (like tapes or portable disks).

The goal of making the data available in real time or near–real time was considered a positive paradigm shift in the atmospheric community, and relaxing this goal by making the data available a few days after receiving it at the primary data system was initially seen by some in the ARM community as not meeting the goal. But over the first 5 yr of the program, the scientific usage pattern of ARM data shifted from near-term comparison with models to an evaluation of a continuous series of case studies. These case studies were usually a series of intermittent days that were specifically suited for the evaluation of a common scientific question. The scientific users typically needed to access a period of data spanning a few weeks to several months. The immediate need to access recent data was generally irrelevant.

So while the scientific need for near-real-time data availability became lower priority, the program maintained the “available within 2 days” approach. This near-real-time approach allowed the infrastructure to identify instrument failures, processing errors, measurement bias, and other data quality issues more quickly (Peppler et al. 2016, chapter 12). This short delay assured that researchers were provided a stable data product with known quality.

e. Data quality

One of the program’s mantras is that “ARM data have known and reasonable quality” (Gracio et al. 1996). Prior to ARM, atmospheric data were typically collected in relatively short duration field experiments, and instrument investigators would have several months after the field campaign to QC their data. ARM’s operational paradigm prohibited this because there was no respite in the data collection for QC. Thus, the program had to develop automated approaches to apply the QC. Three distinctly different approaches were (are) used.

The simplest form of QC applies a set of logical rules based upon the specific data stream and its origin (e.g., the surface radiation measurements at SGP) during the data processing to set a QC flag in the NetCDF data stream file. These QC flags identified values that were too low, too high, too erratic, or failed other expectations.

A higher level of QC could be applied by comparing observations from different sources or by comparing observations to a model simulation where the model was driven by a different observed dataset. The quality measurement experiments (QMEs) were very popular in the ARM Program in the mid-1990s, with many of them developed to help evaluate radiometric instrument data quality (Mlawer and Turner 2016, chapter 14; Peppler et al. 2016, chapter 12). QMEs were automated in the data system, producing another data stream that was analyzed by scientific users and could find more subtle problems with the measurements than could be found using more simple minimum/maximum/delta checks.

Often, the best description of data quality came in the form of manually entered data quality reports (DQRs). These DQRs are more dynamic than QC flags and contain a description of an “event” that altered the normal quality of the data. Events include a wide range of conditions such as instrument degradation and contamination and temporary operating conditions (power failures, frozen or snow covered sensors, etc.). The quality impacts from these events are defined for a specific time range, list of data products, and specific measurements. DQRs are typically submitted by either the instrument mentor [see Cress and Sisterson (2016, chapter 5) for a description] or the Data Quality Office (Peppler et al. 2016, chapter 12). DQRs are now provided as “companion” information with the data files when researchers request data from the archive.

f. Pushing versus pulling data

The initial design of ARM’s data flow expected researchers to define relatively static combinations of data products to be delivered to them. When data files were ready for distribution, the ARM Experiment Center would “push” the data files to the researcher’s system, either over the Internet (via a program like FTP) or by copying data onto computer media and shipping these to the researcher. The “pushing of data” was implemented to meet the initial requirement for near-term comparison between measurements and modeling results and to ensure that the scientific users were getting ARM data. However, the pattern of data use quickly evolved to retrospective analyses of special case study days that occurred during the history of data collection and were relevant to the researcher’s needs. Pushing a steady stream of data to researchers was not necessary for this pattern of use, and this role of the Experiment Center was discontinued. Furthermore, other advancements in Internet accessibility and web applications (see below) made it easier for scientists to select the data they wanted and to “pull” it to their machines. The flow of data to the researchers evolved from a more labor-intensive push to a more efficient and selective pull of data. With the pull method, users only worked with the required data instead of all of it.

g. The growing web and its applications

The rapidly advancing field of web applications, search engines, and the continuing development of the Internet provided new opportunities for the ARM data system to evolve. User interfaces at the ARM Data Archive allowed users to specify dynamic selections for their data needs. These selections created customized lists of data products and data ranges that resulted in data requests from the archive. The archive retrieved the requested data files from the mass storage and put them online for the researchers to download during the next week. Dynamic selections of data from the archive were more adaptable to the evolving needs of the researchers and less labor intensive for the data system staff.

The initial user interface for accessing the data inventory information of the ARM Data Archive required server class hardware and the use of X windows display capabilities on a researcher’s computer. This generally required computers, network capacity, and expertise more substantial than a typical PC and scientist in the early 1990s. The software to interactively display data availability and forms to capture user data requests was tedious to develop and had limited scalability.

The invention of Hypertext Markup Language (HTML) in the early 1990s allowed an unlimited structure of text, pictures, and drawings. The addition of HTML forms enabled users to enter selection criteria into web pages. Developing software to display dynamic content on subsequent web pages based on the user’s criteria was a major boon to ARM. This enabled ARM to communicate to the user manageable portions of its instrument documentation, operations records, data inventory, and data quality information. Web applications and related data retrieval processes enabled more automated processes and allowed for the initial scalability of the ARM Data Archive.

Soon after web technology became available, displaying simple lists or tables summarizing ARM’s scope became more content than most users would absorb. Dynamic web displays of information were very important as the types of instruments moved into tens, the types of data products moved into hundreds, the different measurements across several locations became greater than 1000, and ARM accumulated hundreds of thousands of user accessible files.

The unlimited linkage of web-based information was important for documenting nonroutine operations from the intensive operational period (IOP) and the resulting nonroutine data products. Web links could easily accommodate the layers of detail needed to describe complex combinations of instruments and data products associated with field campaigns.

The web pages were an immediate advantage to ARM and greatly facilitated the creation of large quantities of information. However, they quickly became opportunities for inconsistent organization of information and confusion for ARM users. The ease of creating web information in common word processor software resulted in many people creating content in many different styles with inconsistent keywords and organization structure. In response to this complexity, a staff member was assigned to coordinate web-based information.

The inconsistencies shown in the web information revealed inconsistencies during many steps across the data life cycle. Many other factors led ARM into a period of improved organizational structure (or reorganization). The management, structure, and logic of the data system were also soon reorganized.

h. Dropping operational models from ARM’s scope

In the earliest days, the program had planned on running spectral radiation and single-column models (SCMs) as part of the data system’s infrastructure. However, it quickly became apparent that the challenges of collecting and managing the field measurements required all of the available resources, and thus running SCMs was dropped because the level of effort was too high. However, the spectral radiative transfer models were still run as part of the ongoing QME effort.

a. Consolidation

The early data system management structure of ARM was site-centric, and each field site started independent and parallel implementations. This structure was efficient for getting started, but also resulted in independent designs by each site for data systems, data quality review, and data products. The implementation of multiple solutions to similar problems limited the data system’s ability to keep up with the required growth needed for new sites and instruments. The data systems were a venue where many unnecessary complexities and inconsistencies became apparent from divergent evolution of site data systems and processes. As ARM accumulated more data history than the normal 3–5-yr project, continuing to support the resulting complexity was difficult. Presenting the data collection to the researchers with the archive user interface was problematic.

An infrastructure review committee recommended in 2000 that the overall structure of ARM be changed from site-centric to a structure emphasizing science-centric themes across the array of ARM measurements (ARM 2001). The revised ARM infrastructure included groups with thematic responsibilities (technical coordination, publications and communications, operations, engineering, data quality, data processing, etc.). The new programmatic structure focused common processes for developing site data systems, ARM software, data quality review procedures, and data products across all of the locations. Common solutions enabled the data system to respond more readily to continued growth from new science themes. These changes were applied not only to new instruments and locations, but also to significant redesign of existing processes and products. Variations across the ARM data systems should now only occur to accommodate the environmental and logistical differences between the sites.

Several changes in the organizational structure of ARM impacted the data system. The production-scale processing of ARM data from all sites and VAPs, which was spread over many different locations, was concentrated into a single data management facility (DMF). The DMF succeeded the Experiment Center as a centralized location for ARM software development. The DMF component of the data system also enabled the development of extensive tools for tracking data processing and data flow. The ARM DSView tool provided numerous interrelated reports on processing status and was implemented in 2001 and updated in 2007 and 2011 (Macduff et al. 2007; ARM 2011). The same tool was also used to monitor data system operations at the sites. Data processing at the field sites was limited to the needs for operational oversight and short-term needs for field campaigns. The site data systems and instrument computers were also standardized as they were replaced during this period of ARM history.

The Data Quality Office was established in 2000 to review all ARM data and to address concerns about inconsistent data quality (Peppler et al. 2016, chapter 12), which led to several changes in the management of data quality information. A major update in most of data products in spring 2001 included a standardized structure for QC flags for most measurements. These flags included the results of tests for minimum, maximum, offset limits, and other tests specified by the instrument mentors. The DQRs were expanded to identify specific measurements and data products that were associated with a data quality event. The DQR distribution was restructured from plain text into portable web pages that included a table of contents showing the DQR titles and a tabular format for the DQR details. During this same time period, a new process at the archive began distributing DQRs that retroactively described data quality issues. A customized distribution of DQRs was prepared for each researcher based on their history of requests for data files. Attempts also were made to define a “quality color” (red, yellow, and green) from both the subjective judgment in the DQRs and an automated algorithm based on the prevalence of QC flags. The quality color from DQRs was attainable. Enabling auto quality color with a stable, automated process and comprehendible presentation to the user was elusive.

Other changes from ARM’s reorganization impacted the structure of the overall data system. Gerry Stokes, the first ARM chief scientist, predicted that “all data in the archive would need to be reprocessed seven times” (G. Stokes 1997, personal communication). In the early era, reprocessing was done on the individual data system that originally processed the data (e.g., the specific site data system). However, this was very inefficient and cumbersome. Thus, a reprocessing center was set up to standardize the procedures and records for the reprocessing tasks. Formalized reprocessing enabled ARM to improve data by removing inconsistencies between sites and across time for a single measurement type (e.g., surface meteorology). Reprocessing also applied updated and more robust algorithms to the data and improved the completeness of the data collection.

VAPs are an important source of ARM data. As indicated above, VAPs combine observations from different instruments, sometimes with model output, to create new data streams. Just like the instrumentation in the program, VAP algorithms need to be robust and able to run in all conditions and thus are typically programmed by staff trained in computer science to create robust, well-documented codes. In the early years, these software developers worked directly with ARM principal investigators (PIs) to develop VAPs; however, this was not efficient because often a communication gap existed between the PIs and the developers. Thus, ARM started pairing VAP software developers with “translators”: infrastructure scientists who could help the software developers interpret the scientific algorithms for VAPs and review interim results (Ackerman et al. 2016, chapter 3). The translators regularly coordinated the scope of the VAPs and their priority to improve the consistency of the data collection. Data distribution to all researchers (both ARM scientists and the broader research community) was assigned to the ARM Data Archive. All of these changes improved the efficiency of each of these activities.

b. New approaches to attain consistency

The continually expanding collection of data streams and more powerful and web-based tools to display the ARM data collection revealed yet other challenges for the ARM data system. As the number of data products increased with new locations, instruments, and VAPs, it was increasingly more difficult to maintain consistency for the design of the data products. Detailed aspects of the data product design such as data field names and quality control limits were inconsistent between similar data (e.g., across all of the meteorological or radiometric measurements). This inconsistency was confusing and annoying to the data users. In 2004, efforts began to put all details for the data fields from all data products into a database. This change enabled differences between similar measurements to be readily identified and consistency easier to attain. This database became a production feature of the data system in 2006.

The continually increasing scope of ARM data made it more difficult to determine the details of the inputs required for VAPs and their availability. A database was developed to define the dependencies between the VAPs and their input data streams. This dependency information included the details about data fields needed for input into each resulting VAP data field. This dependency database was added to the data stream configuration information previously described.

Inconsistent ARM language became apparent as the ARM data products began to converge toward consistency. The size and scope of the data collection expanded rapidly in 2000–04 and also contributed to inconsistent language. For example, the thematic categories of measurements used for data discovery at the archive and for the web-based ARM documentation were similar but different enough to be confusing to the researchers. A collaborative effort in early 2006 defined and adopted a metadata structure to be used by both data discovery and web organization. After this metadata consolidation, many of the archive user interfaces and the ARM web pages could be composed dynamically from the common metadata content stored in a single database. These changes assured that ARM users encountered the same language and logic as they interacted with ARM information.

The incompleteness of the metadata was a critical problem when the display of ARM information and access to data became dependent on the common metadata. Throughout this entire period, efforts were made to encourage the creation and recording of metadata about the data products, measurements, instruments, and sites. The initial solution included educating many members of the infrastructure about the scope of metadata and its role in communicating ARM information and data to the researchers. The infrastructure staff contributing metadata ranged from instrument mentors to software developers and translators to data quality staff. The next step included formally defining the metadata elements and developing templates to record and transmit the metadata. Establishing informal processes to referee the consistency of the metadata was the final step. Timely creation of metadata was an elusive objective. ARM shared this challenge with most other research programs during this era.

During this midportion of ARM’s history, the number of data products from field campaigns grew beyond a short list that could be shown on a single web page. The complexity of the data collection from field campaigns was challenging. The structure and scope of field campaign data varied with each campaign. Standardization across the field campaign data was limited because the collaborating researchers used specialized methods for processing and organizing the data. Standardizing the data structures required too many resources. Evolving standards for data structures also limited standardization. Most of the field campaign data followed a consistent structure for each campaign and instrument. More detailed data structures were unconstrained. To accommodate this complexity and retain a maximum amount of these data, the field campaign part of the data system was designed with the following principles:

• When requested, researchers were provided guidance on file names and directory structures.

• A simple FTP procedure or shipped media was used for data transfer from the researchers to ARM.

• The data were organized in a master directory structured by year, site, campaign, instrument, and PI name.

• Documentation for each subdirectory was requested to explain the scope of additional data directories and filenames.

• A web-based user interface was developed that could navigate and provide access to any amount of documentation, complexity, or organizational depth for the data collection.

c. Communicating the scope of ARM’s data collection

The increased size and complexity of the data collection during this period also challenged the researcher’s ability to understand what data were available from the ARM Program over its duration. Simple query logic that searched for the availability of data based on criteria of location, measurement type, and date range failed to find data because the incremental implementation of sites, instruments, and VAPs. The failed queries provided no information about why data were not found (the time range was invalid for a specific location, instrument was offline for repair, etc.). Additional interfaces were developed for the archive that presented hierarchical summaries or catalogs of data availability (McCord et al. 1999).

The expanding data collection also challenged the researcher’s ability to understand the ARM results. Comprehension was limited because scanning the contents of the data files was not possible. Web-based displays showing plots of ARM measurements were developed. In 2004, a user interface (thumbnail browser) went into production at the archive that displayed tens of thumbnails (standardized, postage stamp size, and data plots) per web page. The data plots (both small thumbnails and labeled page-size graphs) were precomputed to assure a responsive and predictable performance of this user interface as the size of the ARM dataset expanded. This user interface allowed researchers to customize displays of specific measurements from multiple data products and visually scan the results in monthly increments (ARM 2004).

A web-based tool for interactively plotting small amounts of NetCDF data (NCVweb) was also developed during the same time period (Bottone and Moore 2003). This NCVweb tool allowed users to select relatively small time ranges of data and “zoom in” on the details of individual measurements. The NCVweb tool was expanded to display the details of the data fields and export small portions of data into text files or smaller NetCDF files. Both the NCVweb and thumbnail browser user interfaces for the archive are still used by researchers. Routinely generated data plots and interactive visualization of data with NCVweb also are used during data quality review [see Peppler et al. (2016, chapter 12) for more details].

The primary themes contained in the midperiod of ARM’s data system history included

• attaining efficiency through the consolidation of data system functions;

• expanding the creation and communication of data quality information;

• converging on consistency in the detailed structure and description for ARM data products and documentation; and

• improving the communications with the researchers through new interfaces to present the scope of the data collection and visualization of ARM’s results.

a. Challenging growth opportunities

Persistent themes for ARM during this most recent period of ARM’s history were growth and accelerating growth. These themes originated from implementing several new sites (e.g., the new permanent site in the Azores) and instruments as well as more tools for efficient data product development. The primary events that triggered growth during this period include the addition of

• a second mobile facility (2008–10);

• the purchase of many new types of instruments with Recovery Act (stimulus) funding (2009–11; Mather and Voyles 2013; Voyles and Mather 2010);

• acquisition of scanning instruments with new types of data products (2010–12);

• implementation of new fixed location in the Azores (2012–14); and

• implementation of a third mobile facility (2012–14).

View Full Size

Cumulative data volume stored in the ARM Data Archive in terms of (top) number of files (millions) and (bottom) volume (terabytes).

Citation: Meteorological Monographs 57, 1; 10.1175/AMSMONOGRAPHS-D-15-0043.1

• Download Figure
• Download figure as PowerPoint slide

The designation of ARM as a national scientific user facility in 2004 (Ackerman et al. 2016, chapter 3) converted ARM from a 10-yr project to a permanent operation for atmospheric measurements, data collection, and data distribution. Performance metrics for the facility were defined for data completeness and increasing the number of users. This designation encouraged ARM to aggressively identify scientific collaborators for mobile facility implementations and other smaller-scale field campaigns. A significant number of new users are added to the ARM user community with each new mobile facility location. All of these factors increased the diversity of data users. The longevity of ARM encouraged researchers to evaluate larger quantities of data. The rate of data usage by the researchers also accelerated (Fig. 11-5).

View Full Size

Cumulative data volume requested from the ARM Data Archive in terms of (top) number of files (millions) and (bottom) volume (terabytes).

Citation: Meteorological Monographs 57, 1; 10.1175/AMSMONOGRAPHS-D-15-0043.1

• Download Figure
• Download figure as PowerPoint slide

As the increase in the outgoing and incoming data volume ranged from 10 to 120 times respectively during this period, the data system required continuing revisions to its design in order to remain nimble. Several changes occurred in the data system to help it reduce the complexity in its design and the chaos in its performance. The changes summarized in the remainder of this section enabled the system to more quickly adapt to the changing requirements.

b. Expansion and coupling of hardware

Expansion for data volume was accommodated primarily with increased data storage at each point in the data life cycle. Data access was increased with high-performance networks and high-performance file servers. The file servers concurrently used several hardware interfaces and numerous storage devices. In 2012, the DMF hardware was moved from Pacific Northwest National Laboratory, which had operated the DMF and its predecessor the Experiment Center from the beginning of the ARM Program, to Oak Ridge National Laboratory. This move enabled the DMF network and storage devices to be combined with the archive. This change enabled quicker adaption and reallocation of the combined hardware capacity to different parts of the data life cycle as needed. In an analogous manner, the site data system has been modified by consolidating individual instrument computers into virtual machines on a single redundant pair of computers with replicated storage. This design reduces maintenance efforts and provides the capability to transition processing from one set of computers to another if the first fails at any of the sites. All of these changes allow the various ebbs and surges of data flow to share common storage buffering resources that are more robust and durable than can be implemented for each process.

c. Consolidation and software efficiency tools

The very large number of new instruments purchased with the Recovery Act funding resulted in the need to build an integrated software development environment (ISDE) as part of the data system. The vision for ISDE included providing standardized tools for the common software tasks. The availability of ISDE would reduce the effort for new software and allow development to focus on the specialized portions of the software. ISDE was also intended to be portable so that software developed by researchers would more readily follow ARM standards for software and data products. After 3 yr of experience with ISDE, its scope was reduced to handling the data input, combining data from multiple inputs into a common interval for sampling rate, data product design specifications (from the existing database), and handling the output. Most of the software that uses ARM data requires some form of these functions. ISDE was renamed the ARM Data Integrator (ADI) to better represent the scope of its functions. ADI was valuable because it reduced the effort for developing new data products (i.e., VAPs). It also focused most of the software development effort on the scientific logic because the reading and writing, as well as the time syncing of different data streams, was now done internally by ADI. ADI also enabled researchers to develop more standardized software and data products when collaborating with ARM.

d. Reducing the complexity

The expansion and requirements for the durability of ARM data processes resulted in the formalization of the VAP development process. In the previous periods of ARM, VAP development could be a very lengthy series of iterations between the translators, the software developers, and the ARM PIs. Earlier styles of VAP development included incomplete processes for defining the data product, selecting advanced algorithms, evaluating the results, and declaring completion of the development. These earlier styles sometimes resulted in endless development processes for VAPs. The formalized process defined a specific series of evaluations for the following:

• confirmation of algorithms to be used and knowledge of their logic;

• decisions to release interim versions of the VAPs data for review by the researchers;

• assessment of the interim progress on the VAP development; and

• creation of documentation of the finalized data product.

The program realized that PI data products offered an excellent way to get additional higher-order processed data into the ARM Data Archive, thus both capturing it and making it available to other scientists. The program created an area in the archive specifically for value-added datasets provided by ARM PIs. With this procedure, a translator worked with the PI to move the data into the archive. Then, a metadata editor was developed allowing the PI to include metadata about the contributed dataset that both described the algorithm used to create the dataset and provided key words so that other PIs could more easily find the dataset within the archive holdings.

The reprocessing of different datasets over time led to multiple versions of data products being used in research projects, especially for those PIs performing long-term analyses. Standards for data products were needed to reduce the complexity of the data processing by the researchers. ARM developed a standards document (and revision process) for data product names and data product design as an extension of the standardization theme in 2012/13. An infrastructure committee representing all portions of the data system, the data life cycle, and a few data users developed these standards. Sustainable standards have enabled the ARM data system to readily expand. They also facilitated the long-term durability of the data products and related software.

Sustaining standards for the names and descriptions of ARM data required improvements for creating and reviewing the metadata. Development of a web application to formally track the proposal, review, and approval of the metadata began in 2012. This application enabled ARM to improve the completeness of the metadata and expand the scope of metadata creation to include the field campaign data. Formalized metadata for these data were needed to consolidate the discovery of routine and field campaign data into a single user interface for the archive. This consolidation has improved the durability of the user interface and helped to expand the researcher’s data discovery.

e. Improving the use of very large ARM datasets

Some of the new instruments acquired in the late 2000s as part of the Recovery Act, such as the scanning cloud radars, Doppler lidars, and aerosol mass spectrometers, create huge amounts of data every day. These files are sufficiently large (many hundreds of gigabytes per day) that transferring these files to PI computers was challenging and very inefficient. To address this, special computers were added to the ARM data system to provide the researchers a means of exploring these new and very large data. These systems (the ARM computing cluster) were implemented in late 2010. They included a cluster of processors, very large amounts of shared memory, and tens of terabytes of storage and workspace. Popular scientific software applications were installed on these systems to facilitate the analysis of these datasets, and dozens of researchers explored the large data products. This addition was another revision to the data system that supported expansion and durability.

Another strategy toward expansion and durability of the data system was making model-ready, condensed, and very specialized (showcase) data products. The development of these showcase data products started in 2007. The processing for showcase data products included new steps such as best-estimate logic that replaced missing values from a primary instrument with the next best choice of an equivalent measurement from a different instrument. Additional quality evaluation and summarization provided the best possible long-term representation of ARM’s core measurements. A showcase, best-estimate product was produced for the full duration of ARM’s history for all sites (Xie et al. 2010). Several data products with extensive summarization, integration, and preparation were expected to further engage the researcher community with ARM data.

The expansion and durability of ARM’s data systems were assured by the consolidation of hardware, software, and development resources into configurations that supported increased flexibility. The consolidation enabled the sharing of common and standardized components. During consolidation, changes occurred to data storage, user interfaces, software development tools, showcase data products, and review processes for interim data products and metadata. All of these revisions contributed to the efficiencies needed for the data system design to adapt to the dramatic growth of ARM’s measurement resources from new instruments at new locations in this most recent period of ARM’s history.