2019 National Centers for Environmental Information Users’ Conference
What: Experts from public, private, and academic sectors met to discuss uses, applications, and requirements of NCEI’s environmental information.
When: 14–15 May 2019
Where: Asheville, North Carolina
NOAA’s National Centers for Environmental Information (NCEI) is committed to developing and producing use-inspired environmental information products that serve every sector of the U.S. economy and support planning and various types of decision-making. NCEI routinely engages with users at hosted events and sectoral conferences to understand and incorporate user needs and requirements into NCEI’s suite of products and services, freely available to the public. User perspectives guide NCEI’s science, research, and product innovation. Requirements derived from user interactions are key to the management and advancement of NCEI’s product suite and often extends into services capabilities, including access and stewardship.
In the spring, NCEI hosted a Users’ Conference in North Carolina to continue its efforts toward understanding user needs. The event provided a forum for users to communicate their requirements and identify ways to make NCEI products and services more usable and useful. NCEI conveyed its current understanding of how its data serve public and private sectors and turned the focus to listening to challenges and successes users experience with NCEI information.
Driven by user information analytics produced over recent years, the 2-day conference focused on five sector- and theme-based sessions including agriculture, retail, private weather service providers, logistics and transportation, finance and insurance, and 1991–2020 Climate Normals with panel speakers from a diverse swath of private and public sectors. The audience represented 14 of 24 economic sectors defined by the Department of Commerce (DOC) U.S. Census Bureau. Panelists presented current work reliant on environmental information, described how they use NCEI data in the context of climate, identified needs, discussed data access and formats, and provided use case examples. Following panel presentations, audience interaction garnered rich, meaningful discussion further highlighting commonalities among user needs.
Participants as a whole clearly recognized and valued NCEI as the nation’s leading authority for environmental information, providing the “gold standard” of data, critical to the day-to-day mechanisms of the nation’s economy. NCEI’s significant responsibility stewarding the most significant environmental information archive in the world, and service to a wide range of users across government, academia, and industry, were repeatedly acknowledged. NCEI provides the necessary foundation that drives a vast value chain supporting commercial applications, policy development, and high-level decision-making across the breadth of the nation’s economic sectors as well as society. Woven throughout the conference dialog, several key requirements themes emerged, spanning various dimensions. The majority of themes were cross sectoral and aligned with current NCEI analytics and industry-/sector-based case study information. Some themes do not fall under NCEI’s purview; however, NCEI is committed to share the feedback with its partners.
Data blending
Data blending faces a host of constraints and challenges such as satellite period of record, availability of data and data types, station-grid interpolation, latency issues, and quality control/quality analysis needs. Such factors increase uncertainty and result in most current blended products being labeled as experimental, contributing to the lingering hesitation to produce blended products for public consumption. NCEI is known as the “gold standard” of environmental data; hence, there is a level of branding jeopardy, or even liability attached to the release of near-experimental products. Participants suggested that NCEI consider producing these types of products with a type of “silver standard” attached, with provision of documentation that addresses uncertainties clearly and transparently accompanied with caveats and warning clauses.
Data relevance
Requests for timelier datasets and product updates were expressed. The International Station Meteorological Climate Summary (ISMCS) also known as “Climate Disk” and the 30-yr normals products were specifically called out in this regard. Logistics companies rely heavily on the Climate Disk, last updated in 1996 and available only on CD-ROM. Users request an update to the Climate Disk made available via an interactive web page. NCEI’s production of the 1991–2020 normals is on track and will continue as planned while also making efforts to respond to the needs heard at the Users’ Conference (especially those repeatedly requested by the energy and construction industries) toward web-based customizable options that enable user-defined normals across varied datasets and variables.
Private sector companies pointed to their worldwide operations consisting of critical day-to-day decision-making requiring consistent and regular global data. Participants recommended NCEI leverage domestic and international partnerships more heavily (e.g., RCC’s, Copernicus) to create a network of standardized, easily discoverable, and accessible datasets to recommend for use.
Data standardization
Standardization of datasets, access mechanisms, and documentation emerged as fundamental needs, with data access noted as one of the greatest challenges. Security hurdles, for example, were cited as a major issue. Many private sector companies as well as government agencies maintain rigid firewalls that prohibit file transfer protocol (FTP) interaction. One panelist described an occasion where they experienced a 3-week delay working around internal security. NCEI must ensure security, privacy, quality, and integrity in data access methods without causing data access bottlenecks. Clearer documentation regarding metadata, dataset structure (including data samples), ingest processes, access methods, and methodology, including data sources and provenance, was requested. Corresponding guidance describing appropriate and recommended use for scientifically sound application was also needed.
Data access
Data discovery difficulties were noted, citing confusion over multiple access locations and inability to navigate data options via FTP. Organizing data more effectively, such as developing sector-specific web pages, could alleviate this issue. A consolidation of data in a manner that makes sense and is straightforward could serve as a one-stop shop for industry-specific users and provide a centralized location for sector- and data-specific documentation. A universal template across all web pages was recommended and should include a standard menu for documentation, appropriate use guidance, alternative datasets, and access links. Access options such as application program interfaces (APIs) should be made more readily available with corresponding use examples, or replicating NCEI’s Regional Climate Center Applied Climate Information System (ACIS) capabilities could facilitate data extraction for bulk access users.
NCEI serves industries across every sector of the U.S. economy; each have unique applications and hence require distinct data, variables, resolution, data coverage, period, frequency, and analyses. The request for provider-side, robust analytic capabilities was repeated frequently throughout the conference. Users prefer to take their analysis to the data rather than taking the data to the analysis, pointing to the basic concept of “data gravity.”
This drives the need for a new centralized data platform. Participants requested that NCEI move toward creating an online tool with a dashboard of input data and scaling options enabling dynamic, custom-tailored datasets. Official datasets are still needed and NCEI should continue to provide these gold standard products (e.g., Climate Normals); however, users yearn to explore data in a manner that better answers their unique questions without cumbersome client-side downloads, subsetting, and data parsing. This further reinforces data standardization requirements to support enhanced interoperability. Unified data formats facilitate dynamic capabilities that users need, like layering datasets and variables and applying analysis with a “click and go” capability.
A majority of participants identified infrastructure tied to or moving to the cloud. Users reported that data extraction processes are required for effective bulk download, forcing a custom build on the client side. If NCEI embraced cloud capabilities and standards, data could live in an optimized cloud environment, allowing direct access for analysis while also ensuring a platform that would support provider-side analytic capabilities and data customization options. The National Environmental Satellite, Data, and Information Service (NESDIS), NCEI’s parent organization, is currently examining possible cloud environments for data storage, computation, and data delivery through its big-data partnerships, and other activities.
Through adjudication approaches for several of the requirements described here, such as a move toward cloud computing, centralized web platform, standardization of data formats and access mechanisms, and data blending, justification and support to produce more highly resolved data will be realized and will be more accessible and usable through a cloud platform.
Data availability
Government shutdown occurrences were also cited as a major issue. Critical data become unavailable, creating difficult and costly situations. Quantifying economic impacts of a government shutdown is not straightforward; however, in the reinsurance world, such disruptions in operations can create cascading effects throughout the industry resulting in premium increases and slow response to claims. This in turn creates conditions that can damage insurance–reinsurance relations, eventually resulting in a disadvantageous environment for consumers. If, for example, NCEI stopped producing daily gridded analysis of temperature and precipitation over the United States, index-based agricultural insurance programs would be at severe risk.
Summary
NCEI has taken great steps over the last several years to improve engagement efforts with industry sectors, requirements gathering and documentation, and adjudication processes. Successes are being realized, and lessons learned are providing guidance to iterate on strategy. These efforts have already led to a greater understanding of user needs and broader and more robust interaction with users and partners.
The 2019 NCEI Users’ Conference furthered these efforts and garnered a wealth of feedback, requirements, and a deepened understanding of environmental information applications. The conference dialog primarily revolved around access improvements, new product development, improvements (see Fig. 1) to existing products, and resolution improvements. Data access discussion pointed to difficulty navigating datasets and around firewalls and the need for user-defined customization options and data analysis capabilities. New product development needs focused on extremes, climatologies, and blended data. Discussion on improving existing products emphasized authoritative data (i.e., uncertainty, precision, quality, and global coverage) and decreased latency. Improvements in resolution spanned nearly all products.
2019 NCEI Users’ Conference documented requirements categorized by classification.
Citation: Bulletin of the American Meteorological Society 101, 5; 10.1175/BAMS-D-19-0323.1
2019 NCEI Users’ Conference documented requirements categorized by classification.
Citation: Bulletin of the American Meteorological Society 101, 5; 10.1175/BAMS-D-19-0323.1
2019 NCEI Users’ Conference documented requirements categorized by classification.
Citation: Bulletin of the American Meteorological Society 101, 5; 10.1175/BAMS-D-19-0323.1
Summarizing further, the overarching requirement defined the need for a straightforward, web-accessible data platform that provides all available data, globally and updated frequently, at the highest possible resolution in a standardized format. The platform should provide user-defined customization options, data layers, and analytic capabilities. Essentially, allow users to analyze NCEI’s data on the fly, apply multiple data types, and select only the ranges needed, bringing the analysis to the data instead of requiring users to find, download, reformat, subset, and merge data types to perform client-side analytics.
Open and free access to robust, resilient, and comprehensive weather and climate data are an absolutely fundamental building block to private and public sectors and vital to economic health. The requirements heard at the 2019 NCEI Users’ Conference will be addressed either directly or indirectly through the NCEI annual product planning and prioritization process. As the next phase of the planning process unfolds, likely to be completed by the time of this publication, NCEI will determine what efforts can be supported through FY20 and those that will be adjudicated in FY21 and beyond. NCEI will continue to support gathering user feedback through mechanisms such as the 2019 Users’ Conference and incorporate documented requirements in future product innovation cycles.
