We are deeply concerned with the trend toward the purchase of commercial weather and climate data by the National Oceanic and Atmospheric Administration (NOAA) and other government agencies that may follow. One major concern is for the stability of both the raw and processed forms of these datasets as commercial enterprises come and go. Also of concern is the loss of free, publicly available data, which provide the backbone for much of the non-profit research at universities, government laboratories, and other nonprofit organizations. Related to this latter point is the questionable practice of private companies gaining the rights over data collected by instruments paid for by United States taxpayers. This letter highlights recent events that prompted our concerns and provides arguments in favor of maintaining an open and free data policy, which has established the United States as the global leader in supporting weather and climate operations, applications, basic research, and education.

A well calibrated dataset with known error characteristics is vital for optimal data assimilation into forecasting and reanalysis models. Satellite-based global navigation satellite system (GNSS) radio occultation (RO) data is a good example of such a dataset, as it is based on a well calibrated timing source and thus can be used to calibrate other datasets assimilated into models, adding to its impact on forecasting skill and the quality of reanalyses used for research (www.wmo.int/pages/prog/www/OSY/Meetings/NWP5_Sedona2012/1b6_Healy.pdf). The stability of RO data also makes them one of the most reliable sources for monitoring long-term trends in the Earth’s temperature throughout the troposphere and lower stratosphere. These data are additionally important for the study and monitoring of space weather. Recently, Congress has requested that NOAA investigate using commercial sources of GNSS RO data as part of the Commercial Weather Data Pilot Program (www.space.commerce.gov/business-with-noaa/commercial-weather-data-pilot-cwdp/). However, recent statements by NOAA at the 2017 COSMIC RO conference indicate that the commercial products are not yet ready for general use (http://spacenews.com/u-s-and-taiwan-cancel-second-set-of-cosmic-2-satellites/). While NOAA and Taiwan’s Ministry of Science and Technology have agreed to drop COSMIC 2B citing funding issues, COSMIC 2A is going forward. In addition, the Weather Research and Forecasting Innovation Act of 2017 authorizes NOAA to continue development of RO technology. Thus, as the technology improves in the private sector, these data are likely to be sought from commercial sources in the near future.

We urge NOAA to remain vigilant and maintain high standards for RO data going forward, given their strong positive impact on weather forecasting, reanalysis products, and climate monitoring. In addition, we urge NOAA to assure the availability of these data at no-cost to the research community, including proper archiving of the raw and processed data for long term scientific studies. Finally, we strongly emphasize the need for a well thought out transition plan, should the RO data processing move from the current COSMIC science team at the University Corporation for Atmospheric Science (UCAR) to NOAA or another commercial provider. These recommendations are consistent with those of the International Radio Occultation Working Group (IROWG: http://irowg.org).

The privatization of data acquisition and processing has, to date, not proven beneficial to the nation’s scientific community. For example, the oil and gas industry note that while realtime and historical lightning data are important to their industry, “the lightning strike data based on surface-based network sensors is proprietary and very expensive for commercial interests to obtain” (https://earsc-portal.eu/display/EO4/C-CORE+1.18+Lightning). Another example is ground based GNSS data, whose processing NOAA privatized in late 2016 as a direct result of advice from the Commercial Weather Data Pilot Program. The raw GNSS signal, together with surface pressure and temperature, can be used to obtain an integrated estimate of the water vapor in the column (PWV), a vital atmospheric quantity important for monitoring water vapor in the atmosphere on all time scales. NOAA now purchases PWV data from a for-profit commercial company, who accesses raw data from ground based GNSS receivers paid for by US and local municipality tax dollars. As of spring 2017, NOAA operational models no longer assimilate ground based GNSS PWV because of the poor quality of the commercial PWV data. However, because the contract did not hold the vendor to a specific level of data quality as a condition for purchase, the government is now in a position of purchasing data it cannot use nor require the company to improve.

Under the contract, NOAA is also not permitted to disseminate these data to the research community. Thus, only the Suominet website (https://suominet.ucar.edu), a government research group funded by the National Science Foundation, currently provides near real-time processed GNSS data in the form of PWV and only for a subset of the available ground based GNSS sites. Because Suominet is a research project, we are not assured of the longevity of this data source. It is also unclear who owns the archival rights to these data going forward, threatening a stable dataset of column water vapor with a long record at many locations throughout the United States. In addition, the practice of making data available only to the operational community and not the research community is divisive and hinders the transition of innovation from research to operations. The lesson learned from the privatization of GNSS data processing is that NOAA must be more diligent when negotiating contracts with private vendors, with contracts including clear guidelines for the quality of the data product as well as some guidelines on how the data will be archived for future research and societal use.

The free and open data policy of the United States sets the standard for data collected around the globe with the purpose of furthering our understanding of the climate system across all time and space scales. The examples above highlight important negative consequences of handing over national observing system assets to private companies or government agencies without a well-thought-out plan for maintaining the free flow of information in support of scientific research and for protecting climate quality data far into the future. Whether data are provided by government or private organizations, the pursuit of scientific knowledge must not be contingent upon the commercial value of the data needed for that activity. Over time, the private collection and processing of data by for-profit companies will lead to the separation of those with expertise in the datasets from those that use those data for scientific research. This, together with the division of the research and operations communities through exclusive contracts between private companies and the weather service, will greatly hinder scientific progress and innovation in the United States. To maintain scientific advancement, data must be freely available to researchers and educators in both their raw format, as well as processed into research quality datasets, such that both experts and non-experts in the acquisition of the data can make use of them. We are certain that if the cost of dividing up the scientific community into observational, research and operational components were factored into estimates of scientific data collection, a free and open data policy would prevail as the best way to maximize the value of our national investment in those data and their impact on scientific research and education.