Abstract

High quality data sources are critical to scientists, engineers, and decision makers alike. The models that scientists develop and test with quality-assured data eventually become used by a wider community, from policy makers’ long-term strategies based upon weather and climate predictions to emergency managers’ decisions to deploy response crews. The process of developing high quality data in one network, the Oklahoma Mesonetwork (Mesonet) is detailed in this manuscript.

The Oklahoma Mesonet quality-assurance procedures consist of four principal components: an instrument laboratory, field visits, automated computer routines, and manual inspection. The instrument laboratory ensures that all sensors that are deployed in the network measure up to high standards established by the Mesonet Steering Committee. Routine and emergency field visits provide a manual inspection of the performance of the sensors and replacement as necessary. Automated computer routines monitor data each day, set data flags as appropriate, and alert personnel of potential errors in the data. Manual inspection provides human judgment to the process, catching subtle errors that automated techniques may miss.

The quality-assurance (QA) process is tied together through efficient communication links. A QA manager serves as the conduit through whom all questions concerning data quality flow. The QA manager receives daily reports from the automated system, issues trouble tickets to guide the technicians in the field, and issues summary reports to the broader community of data users. Technicians and other Mesonet staff remain in contact through cellular communications, pagers, and the World Wide Web. Together, these means of communication provide a seamless system: from identifying suspicious data, to field investigations, to feedback on action taken by the technician.

1. Introduction

The Oklahoma Mesonetwork (Mesonet), developed through a partnership between the University of Oklahoma and Oklahoma State University, is a permanent mesoscale weather observation network. Care was taken along every step of the process to ensure that the Oklahoma Mesonet would provide research-quality data. The procedures documented in this manuscript are designed to ensure this quality, from the earliest planning stages of the network through operational data monitoring and long-term analyses. This manuscript details quality assurance (QA) procedures developed through the course of building the Mesonet and employed operationally in May 1999.

The Oklahoma Mesonet operates 115 stations on a continuous basis (Fig. 1). Thirteen atmospheric and subsurface variables (hereafter, parameters) are recorded every 5 min at each site, producing 288 observations of each parameter per station per day (Elliott et al. 1994;Brock et al. 1995). Several other parameters are observed every 15 or 30 min. From its commissioning in March 1994 through May 1999, the Oklahoma Mesonet has successfully collected and archived 99.9% of over 75 million possible observations. Because of this continuous observation cycle, a need existed to ensure the quality of data coming from over 2500 instruments. A comprehensive QA system was developed to complement the network’s efficient collection and transmission of environmental observations. The system utilizes feedback from an instrumentation laboratory, field comparisons, automated tests, and visual analyses to recognize and catalog suspect and/or erroneous observations. Efficient communication between all components in the QA system is essential to quickly replace questionable instruments.

Fig. 1.

Map of Mesonet site locations. Land ownership for each of the sites is according to the symbols defined in the legend. There are 42 sites located in the ARS Micronet in southwest Oklahoma.

Fig. 1.

Map of Mesonet site locations. Land ownership for each of the sites is according to the symbols defined in the legend. There are 42 sites located in the ARS Micronet in southwest Oklahoma.

2. Network design considerations

The Mesonet Steering Committee established 11 subcommittees, each drawing upon the experience of experts within their respective fields. Seven of these subcommittees (see appendix) offered recommendations pertaining to the quality of data recorded within the network. The subcommittees represented three focus areas. The Site Standards and Site Selection committees developed a consistent set of criteria for site selection and provided guidance in locating sites. The Parameter Selection and Sensor Specification committees evaluated many diverse requests for instrumentation to be installed on the Mesonet towers and developed recommendations for instrument purchases. The Station Maintenance, Quality Assurance, and Data Management committees developed guidelines to maintain the quality of and access to data once the network was established.

The Site Standards Committee recommended that sites should be located in rural areas, representative of as large an area as possible, and flat, with all obstacles being at a distance of more than 300 m away from the wind sensors (Shafer et al. 1993). These guidelines were followed closely; however, in a few cases, some guidelines could not be met. These cases resulted from 1) the nature of the terrain; 2) a lack of suitable sites offered;3) the a priori decision to locate on particular parcels of land, such as existing agricultural research stations. Site photographs and documentation are available online at http://okmesonet.ocs.ou.edu/ so data users can “visit” a site.

The Site Standards Committee also provided guidance for site layout and parameter selection. Wind speed and direction are measured at a height of 10 m to match World Meteorological Organization (WMO) standards, and temperature and relative humidity are measured at 1.5 m for consistency with existing National Oceanic and Atmospheric Administration cooperative observations and airport stations. The characteristics of Mesonet sites are also generally consistent with standards recommended by the American Association of State Climatologists (AASC) for automated weather station networks. The layout for all Mesonet sites is depicted in Fig. 2. The tower stands nearly in the center of a 10 m × 10 m enclosure. It is surrounded by a cattle-panel fence, 1.3 m high, to secure the area from animals and nearby human activity.

Fig. 2.

Schematic of Mesonet instruments site layout. The perspective is a side view of the station looking northward.

Fig. 2.

Schematic of Mesonet instruments site layout. The perspective is a side view of the station looking northward.

One difference between Mesonet and AASC recommendations is in the height of the wind monitor. Meyer and Hubbard (1992) note that the AASC’s recommended height of 3 m for wind measurements is a compromise between the expense of installing 10-m towers and problems with exposure affecting wind measurements at lower heights. Because the Site Standards Committee recommended installation of 10-m towers for wind measurements, the AASC’s concerns for exposure are mitigated. A second difference concerns the height at which the rain gauge is mounted. The AASC recommends a height of 1.0 m to reduce splash effects, while the WMO recommends a height of 0.3 m to reduce wind effects. The AASC’s concerns of splash effects were resolved by modifications to the rain gauge made by Mesonet staff (discussed in section 4b). Installation of wind screens around the rain gauges to reduce turbulence in the vicinity of the gauge orifice addresses the WMO concerns.

In addition to general guidelines for the network, the Site Selection Committee recommended that several sites be provided to allow comparison with other instruments and other networks. Two of the original 108 sites, representing different climate regimes, were installed within 100 m of Automated Surface Observing System sites (located at Hobart and McAlester). These collocated sites provide an opportunity to compare observations from two networks in different climatic zones (Crawford et al. 1995). Eleven additional sites were located within 2 km of a National Weather Service Cooperative Observer Site.

An intercomparison site in Norman, located 100 m from an operational Mesonet site, allows in-field comparison of data from different instruments. Higher quality instruments, whose cost prohibits their deployment throughout the network, may be temporarily deployed at the intercomparison site as part of various research projects. The intercomparison site also allows evaluation of proposed instrument changes, the addition of new instruments, and changes in network configuration, without affecting data collected from operational sites.

The Sensor Specification Committee developed performance specifications for each instrument individually, designing the specifications to meet criteria for both research and operational purposes. The make and model of each instrument were selected separately for each parameter, allowing uniformity among those sensors deployed in the field. By equipping operational sites with similar sensors, the potential of measurement bias, when comparing measurements between sites, is reduced. This strategy also allows technicians to draw from a common stock of spare sensors when sensor replacement is required. By using similar instruments at all field sites, a user does not need to be concerned by different instrument error characteristics when comparing data between sites.

The remaining committees provided guidance relating to collection and archival of data. In particular, the Quality Assurance Committee recommended the following steps to develop a quality data stream:

  • laboratory calibrations to test sensors at delivery and at routine intervals thereafter;

  • field intercomparisons, both during technician visits and through site intercomparisons;

  • real-time, automated data-monitoring software;

  • documentation of sites and processes;

  • independent review; and

  • publication of data quality assessment

While not all recommendations have been fully implemented, the following sections document steps taken toward meeting these recommendations.

3. Overview of Mesonet QA processes

The need for thorough and efficient data archival and retrieval, combined with the need to optimize daily performance of the network, dictated a dual purpose for the Oklahoma Mesonet’s QA system:

  1. augment data archives with a trustworthy assessment of the confidence in each datum, and

  2. assess the ongoing performance of the network to keep current and future data quality at the highest level that is operationally possible.

Fortunately, the techniques that apply to these goals overlap, and the same QA tools can be used to assess both past and present data quality. The real-time nature of the Oklahoma Mesonet requires more than post facto analysis of data collected during field projects (e.g., see Wade 1987). The techniques must apply to ongoing data collection to identify problems before they become serious.

The Oklahoma Mesonet’s QA system compiles information from four distinct analysis classes: laboratory calibration and testing, on-site intercomparison, automated routines, and manual inspection. Each produces valuable information about the network’s performance. The results from any one component are shared throughout the system to establish an accurate assessment of data quality.

Because of the volume of QA information and the need for coordinated QA decision making, a QA manager is employed to act as a “traffic cop” for instrument and data-quality issues. The QA manager maintains communication among all facets of the QA system, issues calls-to-action regarding problem sensors, tracks and archives specific instrument/data problems, and guides further development of the QA system.

These quality-assurance procedures integrate into a seamless system that incorporates both time and system components. Table 1 depicts the quality-assurance process on timescales that range from when the data are recorded to a long-term investigation of data quality. Quality-assurance procedures begin at the site. Laboratory-verified calibrations and sensor rotations ensure that quality instruments are deployed. Codes may also be transmitted from a site to indicate potential datalogger problems or the presence of a technician at a site.

Table 1.

Timescales of quality-assurance procedures used by the Oklahoma Mesonet.

Timescales of quality-assurance procedures used by the Oklahoma Mesonet.
Timescales of quality-assurance procedures used by the Oklahoma Mesonet.

Computers at the base station in Norman monitor system operations and verify that all possible data have been collected. If data are missing, the computers automatically and repeatedly attempt to contact the station until data are retrieved. As data are collected, they are sent to an archiving computer that verifies that observations are within acceptable bounds. The archiving computer rechecks that there are no gaps in the data records.

Throughout the day, data users, Mesonet staff, and the QA manager monitor the data to make sure they are collected and that system performance is acceptable. Obvious erroneous values reported by a sensor may be noted in “snapshot” images of data from the network or in time series graphs of reported data from a single station.

Each night, automated QA procedures check the data for more subtle errors and produce a report for the QA manager, which is waiting in the manager’s e-mail “in basket” the next morning. The QA manager scans the report, makes determinations of potential problems, and issues trouble tickets as necessary. Trouble tickets may be issued at any time, but the majority of trouble tickets result from the morning QA reports and from monthly assessments of network performance.

Every month, visual inspection of the data archives yields clues to more subtle effects, such as sensor bias and drift. At the end of the month, these analyses are assembled into the QA manager’s report, which often spur more requests for technicians to investigate sensors at certain sites. Several times every year, technicians visit each Mesonet site, perform in situ analyses and intercomparisons, and repair or replace defective sensors.

Testing and deploying quality instruments are crucial parts of ensuring quality data. The efforts of technicians and instrument specialists in the Mesonet’s calibration laboratory provide a long-term foundation for network performance. Ongoing calibration of all sensors and field rotation represents one portion of instrument quality. A second part is an examination of instrument performance and a comparison of data from similar instruments. Recommendations for instrument design changes, which may result from these efforts, feed back into the system to enhance the long-term quality of data from the Oklahoma Mesonet.

Data are never altered. Instead, all archived Mesonet data are coupled with QA “flags” [“good,” “suspect,” “warning,” or “failure” (QA flags 0, 1, 2, and 3, respectively; see Table 2)] that indicate the level of confidence that Mesonet personnel place upon each observation. The use of four values to characterize the relative quality of data provides more information to the data user regarding confidence in the observation than would a dichotomous flag (0, 1). Based on user preference, the QA flags can be applied during data retrieval (eliminating varying degrees of suspicious data), or the flags can be written separately. The former option offers convenience, while the latter allows the user to make the ultimate decisions about data quality. Using this information, data users can make informed decisions about the application of Mesonet data to their projects without the limitations of data that have been altered before being seen by the users.

Table 2.

Mesonet QA flags with brief descriptions. These flags are available with all archived Mesonet datasets.

Mesonet QA flags with brief descriptions. These flags are available with all archived Mesonet datasets.
Mesonet QA flags with brief descriptions. These flags are available with all archived Mesonet datasets.

The use of quality-assurance flags that accompany the data is analogous to the procedures described by Meek and Hatfield (1994) and Snyder and Pruitt (1992). Meek and Hatfield employ QA flags, which supplement but do not alter the data, to describe which test the data failed. Also, consistent with Meek and Hatfield, QA flags are used as a supplement rather than a substitute for regular field maintenance and calibration. Snyder and Pruitt use flags slightly differently, with QA flags being grouped into two general categories: informative and severe. Their severe flags correspond to missing data or inoperative sensors (our flags 9 and 8, respectively) or data out of range (our 3). The informative flags include data derived from other variables or aggregated data that contains one or more flagged hourly observations. They also include flags for data slightly out of range, corresponding to our suspect category, or data that could not be tested. The Mesonet does not include the not tested category because all data undergo at least a range check.

Several components of the QA system refer to a qualparm table. This table is the Mesonet’s living history of network performance and data confidence. Each column represents a specific Mesonet parameter; each row signifies a change in the QA status of one or more parameters at a specific time (see example in Fig. 3). The numerical values are synonymous with those of the Mesonet QA flags given in Table 2.

Fig. 3.

A portion of the Mesonet qualparm table, detailing events at the El Reno (ELRE) site. On the afternoon of 21 Nov 1995, a grass fire damaged or destroyed several instruments; this event is captured on the fourth line of the ELRE section. The eighth line shows a barometer failure that occurred in Aug 1997. The column headers are, in order, station number (STNM), station identifier (STID), series of flags for 5-min resolution parameters, series of flags for 15-min resolution parameters, and date/time of change. Comments may be retained to the right of the date/time.

Fig. 3.

A portion of the Mesonet qualparm table, detailing events at the El Reno (ELRE) site. On the afternoon of 21 Nov 1995, a grass fire damaged or destroyed several instruments; this event is captured on the fourth line of the ELRE section. The eighth line shows a barometer failure that occurred in Aug 1997. The column headers are, in order, station number (STNM), station identifier (STID), series of flags for 5-min resolution parameters, series of flags for 15-min resolution parameters, and date/time of change. Comments may be retained to the right of the date/time.

Because the qualparm table contains both historical and current information, it acts as a database of past network performance and a baseline for current QA processes. The table is manually produced and edited, which provides the opportunity to incorporate human judgment within automated QA processes. In addition to specifying errors in data, the qualparm table can be used to override decisions made by the automated QA (see flags 5 and 6 in Table 2). It has a strong influence on, but does not always dictate, the final QA flags assigned to each Mesonet observation.

4. Instrument calibration and comparison

The Oklahoma Mesonet operates a full-time laboratory calibration and field maintenance facility, staffed by an instrumentation specialist, a meteorological engineering technician, and five full-time electronics technicians. The laboratory provides a facility where Mesonet personnel can verify the calibration of sensors, compare instrument performance to other reference instruments, and test instrument and network design improvements.

The instrument laboratory is designed for two purposes. First, every sensor installed in the Mesonet must pass various calibration processes. The Mesonet’s instrument laboratory provides an opportunity for independent verification of the manufacturer’s calibration. No sensor is deployed to a field site without this verification. Second, the instrument laboratory provides feedback on the performance of instruments and suggests improvements for the network. The controlled environment of the laboratory and field tests from the Norman reference station provide side-by-side comparisons of reference and operational instrumentation. Technicians also test the effects of solar radiation, wind, temperature, and other atmospheric conditions on instruments, with the goal of improving instrument design.

a. Instrument calibration

Each Mesonet sensor undergoes a verification of its calibration coefficients at several points along its operational life cycle. The cycle consists of 1) a predeployment laboratory check of the sensor calibration (“pre-cal”), 2) an on-site instrument intercomparison when the sensor is installed, 3) periodic on-site intercomparisons during normal operation, and 4) a postfield laboratory check of the sensor calibration (“post-cal”). Sensors may be removed from the field because either a problem is detected or a sensor approaches a recommended “site-residence time.”

Tests in the laboratory utilize high quality reference instruments (Richardson 1995). These are used to verify that all sensors are within established specific inaccuracy limits for each instrument deployed in the Mesonet (Brock et al. 1995). The pre-cal verifies that each new sensor performs within these limits before it is deployed to a field site. Calibration coefficients, which are determined in the laboratory for rain gauges, pyranometers, and soil moisture sensors, are applied to these data as they are retrieved from the Mesonet archives. Because the coefficients are not programmed into the data loggers, the data logger code does not need to be revised with new coefficients when a sensor is replaced. In addition, this technique permits the adjustment of postprocessed data on a case-by-case basis, such as may be required to correct errors in calibration coefficients or adjust for sensor drift, without altering data stored in the archives.

Sensors remain in the field until problems are encountered, design upgrades are needed, or the sensor approaches its recommended site-residence time. Design modifications are needed, for example, if a design flaw is discovered, or if a manufacturer recommends an upgrade to its instruments. The sensor’s residence time is the period beyond which a sensor typically begins to show symptoms of excessive drift. The limit is established based upon network experience with that type of instrument and manufacturer recommendations. Upon reaching the limit, the sensor is “rotated” back to the calibration laboratory for recertification and recalibration as opportunity and spare units become available.

Before any adjustments are made to a sensor on its return to the laboratory, the post-cal check attempts to determine how well that sensor was performing at the time of its removal from a site. The sensor is then cleaned and any repairs and modifications are made. At this point, a series of pre-cal tests and adjustments are employed until the sensor’s performance falls within established inaccuracy limits. Results of pre-cal checks, instrument upgrades, and post-cal checks are recorded in a database for use by technicians and by the QA manager if questions arise about a specific sensor’s performance. Ultimately, the sensor is reinstalled at a Mesonet site at the next opportunity and the residence time starts again.

b. Instrument design tests

The knowledge gained from the calibration laboratory and from tests using the reference station is used to shape instrument research and design upgrades to improve performance. Aside from direct instrument evaluations, field studies, such as examining heat flux, have also contributed new insights into the performance of the Mesonet. The following descriptions show how laboratory research on instrument and system designs have contributed to an improvement in instrument performance, and thus, to the quality of data recorded from the Mesonet.

1) Rain gauge retrofit

An early “problem instrument” was the rain gauge (MetOne 099M tipping bucket with 0.25-mm resolution). By early 1994, serious shortcomings in the rain gauge’s performance became evident, necessitating modifications to the gauge design. Technicians diagnosed significant splash effects and recommended several alterations, including replacing the perforated plate screen inside the catch funnel with a wire mesh screen and increasing the height of the catch funnel sides by 7.6 cm. These alteration criteria were shared with the manufacturer and modifications were made to the gauge design.

Experience in the field showed additional problems with the tipping bucket and its bearings. Heating and cooling appeared to crack the bearings, dislocating buckets on some gauges. In addition, separation of the mercury in the tip switches caused problems of either apparent under tipping or double tipping. Technicians recommended redesigning the bearings, resoldering weak bucket seams, drilling “weep holes” on the underside of the bucket dividers, reshaping the bucket pivot, and changing from mercury switches to magnetic reed switches. These modifications were performed by Mesonet staff in the calibration laboratory. Many of these modifications were also adopted by the manufacturer.

2) Radiation shield design studies

Richardson et al. (1999) studied the limitations of naturally ventilated solar radiation temperature shields. They showed that the characteristics of the temperature sensor were important in determining radiation heating effects (e.g., heating from solar radiation). Gill nonaspirated, multiplate radiation temperature shields (Gill 1979) are designed to protect air temperature sensors from solar radiation, but this may not be an optimal design for all temperature sensors. If the temperature sensor is highly reflective and has a small diameter, then a radiation shield that maximizes flow past the sensor is preferable. Such findings result in improved measurements when new instruments are chosen for system upgrades.

3) Temperature and relative humidity sensing problem

The Mesonet routinely measures temperature at 1.5 m using a Campbell/Vaisala HMP35C temperature and relative humidity sensor and at 9 m using a Thermometrics Thermistor (TMM). For a study of heat flux, Brotzge and Crawford (2000) installed a TMM sensor at 1.5 m to minimize the influence of sensor characteristics on measurements between the two levels. During this study, the data from the TMM at 1.5 m was also compared with data from the existing HMP35C (Fredrickson et al. 1998). Temperature differences that ranged from 0.5° to nearly 2.0°C were discovered at eight of the nine test sites.

These temperature differences were consistent with those that would be expected from errors in sensor calibration. However, precalibration records, on-site intercomparisons, and other QA efforts indicated that all sensors were performing within expectations. Because the TMM sensors had recently been calibrated and installed, the HMP35C sensors from several sites were returned to the laboratory for additional calibration checks. These units again indicated no significant calibration problems. Reference sensors and calibration records were also reviewed to eliminate these as possible sources of errors.

Because no errors could be found in the instrumentation or calibration records, technicians then conducted tests on the datalogger program code. They discovered that the errors in the HMP35C sensors resulted from measuring relative humidity before measuring temperature, for this particular combination of probe and logger code. Residual voltage from the relative humidity sensor circuit superimposed an extraneous signal upon the voltage from the temperature sensor. When temperature was sampled before relative humidity, no such effects were observed. The problem was not discovered earlier in the calibration process because the laboratory calibration program sampled temperature first, whereas all field and comparison test kits sampled humidity first. Thus, a minor modification to the datalogger’s program code solved a problem that appeared to be attributable to sensor calibration errors.

5. Site visits: Intercomparison and maintenance

Field tests are performed by Mesonet technicians, each of whom is responsible for specific Mesonet sites. Each site is visited at least three times annually for general maintenance and on-site sensor intercomparisons. Although this maintenance interval is less than most automated weather station networks (Meyer and Hubbard 1992), daily monitoring of data from the network makes this routine maintenance interval adequate to maintain the quality of the sensors at the field sites. Because of this daily monitoring, problems are detected quickly, and technicians may be dispatched for additional site visits as necessary.

General maintenance consists of evaluating the overall integrity of the site (e.g., structures, guy wires, adverse weathering) and trimming vegetation. Stanhill (1992) demonstrated the importance of frequently cleaning the outer hemisphere of the pyranometer. Extended periods between cleaning can result in as much as an 8.5% reduction in net solar radiation due to the deposition of dust and particles on the dome. Because of concerns about site security and integrity, only Mesonet technicians are allowed within the site enclosure. More frequent maintenance of general site characteristics, such as trimming grass and cleaning the pyranometer dome raised concerns that inexperienced local personnel may disrupt soil temperature plots, accidentally loosen or cut wires, or dent the pyranometer dome during cleaning. In addition, with more than half the sites located on privately owned land, weekly visits to sites could strain relations with the site owners. Consequently, detection of underreporting pyranometers between technician site visits is left up to the Mesonet QA manager.

On-site sensor intercomparisons use a set of reference instruments identical to those at the site. These are typically performed for air temperature, relative humidity, barometric pressure, and solar radiation. The reference instruments are frequently recalibrated in the laboratory. The intercomparisons are performed using software that records data from both the operational and reference instruments in 1-min intervals. The technician subjectively selects a quiescent 5-min interval for calculation of average sample error (Fig. 4). This minimizes the potential for meteorological transients to cause the technician to misdiagnose instrument performance, which could be more likely if only a single pair of observations were used. By using on-site intercomparisons each time a technician visits a site, the need for duplicate humidity sensors as recommended by Allen (1996) is mitigated.

Fig. 4.

Display from on-site intercomparison software. Bottom panel: A time series of relative humidity observations from the operational instrument (dotted line) and the reference instrument (solid line). Top panel: The absolute error between the two sets of observations. Acceptable error bounds are indicated by the horizontal dotted lines. As a result of errors exceeding allowable bounds, the technician replaced the sensor.

Fig. 4.

Display from on-site intercomparison software. Bottom panel: A time series of relative humidity observations from the operational instrument (dotted line) and the reference instrument (solid line). Top panel: The absolute error between the two sets of observations. Acceptable error bounds are indicated by the horizontal dotted lines. As a result of errors exceeding allowable bounds, the technician replaced the sensor.

The technician physically checks other sensors for obvious problems, either visually or audibly (e.g., listening for noisy bearings in the wind monitor). To test the rain gauge, the technician drips a known quantity of water from a specially designed reservoir into the gauge at a known constant rate. The gauge’s response is compared to its calibration data from the laboratory records. The technician then partially disassembles the gauge, visually inspects, cleans, and reassembles it, and repeats the calibration test.

These on-site intercomparisons are not used to correct data. Instead, they provide performance checkpoints during the sensor’s operational life cycle. As a result of these visits, technicians may detect sensor problems that are difficult to ascertain by other components of the QA system. On the other hand, a site visit may reveal that a perceived QA problem is attributable to a real local effect.

During the time that a technician is present at a site, a special data flag is activated in the site datalogger. The flag is reported with the regular observations and is archived for future reference. Should questions arise concerning any aspect of data quality, it is possible to check the archives of QA flags to see if the presence of a technician could be a proximate cause of the suspicious data.

6. Automated quality assurance

During each overnight period, a series of automated QA techniques are applied to the previous day’s data. Each observation is checked for validity by five separate test routines: range, step, persistence, spatial, and like-instrument comparison. The first three of these tests are similar to those described by Meek and Hatfield’s (1994) bounded values, rate-of-change, and continual no-observed-change tests. Results from each test are then passed to a decision-making algorithm that incorporates the results from all tests into a single QA flag, similar to the Complex Quality Control method described by Gandin (1988). The resultant flag is then compared with the corresponding flag from the qualparm table (see section 3). The larger of these two values is retained and written to the daily QA files.

a. Automated QA routines

The range test is based upon a combination of performance specifications for each sensor and the annual climate extremes across Oklahoma. Each parameter has predetermined limits (Table 3). Any observation that occurs outside of the maximum or minimum allowable value is flagged as a “failure.” The range test is dichotomous and is the only test capable of indicating a failure by itself.

Table 3.

List of Mesonet parameters and threshold values for the various QA tests. A value of −999 indicates a parameter is not checked by that test. Thresholds for wind speed and soil temperatures for the like-instrument comparison test have not been determined; these tests are not operational.

List of Mesonet parameters and threshold values for the various QA tests. A value of −999 indicates a parameter is not checked by that test. Thresholds for wind speed and soil temperatures for the like-instrument comparison test have not been determined; these tests are not operational.
List of Mesonet parameters and threshold values for the various QA tests. A value of −999 indicates a parameter is not checked by that test. Thresholds for wind speed and soil temperatures for the like-instrument comparison test have not been determined; these tests are not operational.

The step test compares the change between successive observations. If the difference exceeds an allowed value, distinct for each parameter, the observation is flagged as a “warning.” If either one of the data points used in the comparison are missing, the test indicates a null result for those pairs; other tests determine which data point is missing. The step test has proven useful for detecting erroneous readings due to loose wires or datalogger problems.

The persistence test checks an entire day’s data from a single station, one parameter at a time. The mean and standard deviation for that parameter are calculated. If the standard deviation is below an acceptable minimum, the corresponding data values are flagged as either “suspect” or warning, depending upon the departure of the standard deviation from the minimum threshold. The persistence test, because it uses aggregate statistics, cannot discern which observations within the time period are responsible for the offense. Consequently, all data values for that particular site and parameter receive the same suspect or warning flag.

The persistence routine also uses a “delta test” to check the largest difference between any pair of observations within a selected time range, usually 24 h. If the difference is less than a minimum acceptable change, all data values for that parameter for the station under consideration are flagged as suspect or warning, depending upon the departure of the delta from the threshold.

The more serious of the two flags from the persistence tests is retained and reported as the resultant flag by the persistence routine. The persistence test is useful for determining damaged instruments or those “stuck” at a particular reading (e.g., due to icing conditions or faulty communications between an instrument and the datalogger).

The spatial test utilizes a one-pass Barnes objective analysis routine (Barnes 1964) to estimate a value for each observation. The test is designed to detect gross errors in individual observations; more subtle errors are identified through manual inspection. Thus a one-pass Barnes analysis provides reasonable estimates without placing too many demands upon computing resources.

Observations are weighted according to their distance from the station being evaluated:

 
formula

where Ze is the estimated value of a parameter at a particular station, zi is each observation, and w is the weight applied to the observed value, based on the distance between the observation and the point being estimated (ri). The weight decreases exponentially with distance from the station:

 
w(ri) = er2i/ko.

The weight function ko is determined by the Barnes routine, based upon the mean station spacing within the network. The radius of influence is approximately 100 km for the Oklahoma Mesonet.

All stations within the radius of influence, except the station being evaluated and those stations identified as failures from the range test or as warnings or failures in the qualparm table, are used to calculate an estimated value. The mean, median, and standard deviation of the observations within the radius of influence are also calculated. In the central part of the Mesonet, 20–25 stations are typically used to determine the estimates. The difference between the observed and estimated value is compared to the standard deviation:

 
formula

Any observation whose difference exceeds twice the standard deviation (Δ > 2) is flagged as suspect; any difference that exceeds three times the standard deviation is flagged as warning. If fewer than six observations were used to determine the estimated value, no flag is set. Thus, stations in the Oklahoma panhandle are not included in the spatial analysis.

Standard deviations are used rather than absolute thresholds to allow increased variability during situations of large contrasts across the network (K. Brewster 1995, personal communication). This technique tolerates larger departures from estimated values for stations along a frontal boundary than for those stations in a nearly uniform air mass. To avoid flagging stations when the calculated standard deviation is small, a predetermined threshold for the standard deviation is used (Table 3). For example, if the standard deviation of air temperature for a subset of sites is 0.5°C, stations departing by more than 1.0°C from the estimated value would be flagged; however, a minimum standard deviation of 3.0°C ensures that only stations differing by more than 6.0°C are flagged. If a standard deviation of 5.0°C is noted across a frontal boundary, stations would have to depart from their estimated values by more than 10.0°C before being flagged. Although this procedure reduces the number of false alarms, small errors are difficult to detect under such circumstances.

The like-instrument test compares pairs of similar parameters, for example air temperature at 1.5 and 9.0 m. Any differences exceeding a specified threshold are flagged as suspect; differences more than twice the threshold are flagged as warnings. If one parameter in the like-instrument pair is bad, the like-instrument test cannot discern which observation is the culprit; both observations receive the flag. The final determination as to which one is bad is made in the decision-making algorithm. The like-instrument test also is used to identify good data, which are flagged by other routines, such as instances of localized temperature inversions. Thresholds were determined via a climatological study of temperature inversions across Oklahoma (Fiebrich and Crawford 1998).

The determinations of each of these routines are sent to a decision-making algorithm. The algorithm first flags all missing observations as “missing data” in the final QA archive. The algorithm also notes those stations at which certain sensors are not installed and marks those observations with a flag of “never installed” in the QA archive.

Corrections are made to flags from the step test by combining them with the results from the spatial test. The step test sets a flag if the data series marks an abrupt change. It cannot discern whether this change is toward or away from “background” values. By combining the results with the spatial test, the step away from background can be increased in severity, while the one returning toward background can be decreased. The step and spatial tests combined yield a failure if an observation is flagged by both tests, or a suspect flag if it is flagged only by the step test. If the datum is flagged only by the spatial test, the spatial test flag is retained.

Flags from the like-instrument test are compared with the corresponding spatial test to determine which observation of the pair is questionable. If an observation is flagged by both the like-instrument and spatial tests, the final determination will be either a warning, if both routines indicate the observation is suspect, or a failure if one of the two routines indicates a more serious error. If an observation, flagged by the like-instrument test, exhibits spatial consistency with neighboring sites, it is considered good.

The like-instrument test is also used to compensate for shortcomings in the spatial test. Observations that pass the like-instrument test exhibit a consistency between the two parameters checked. If one of these observations is flagged by the spatial test, the cause may be attributable to a real, but localized, effect, such as a temperature inversion. To compensate for such effects, the spatial flag is downgraded to the next lower level in such instances.

After these adjustments have been made, the results from individual tests are added together and then compared to the qualparm table value. If the qualparm value is five or six, the corresponding QA flag is either downgraded by one level or reset to zero (flags 5 and 6, respectively). For all other qualparm values, the greater of the QA flag or qualparm is retained in the QA archive.

b. Using automated QA summary reports

Along with individual flags, the automated QA produces a summary report each morning (Fig. 5) that details data irregularities from the previous day. The report is automatically e-mailed to the QA manager, who scans it for potential new instrument problems and checks suspicious data to determine whether further action is warranted. An entry is included in a list of “flag counts” if more than 10% of a parameter’s observations from a station for that day are indicated as suspect, or if one or more observations are listed as a warning or failure. The 10% suspect threshold prevents the QA manager from unnecessarily examining spurious data, allowing concentration on more serious problems.

Fig. 5.

An excerpt from the automated QA summary report for 21 Sep 1997. The letters in parentheses represent the current sensor status in the qualparm table (S, suspect; W, warning; F, failure), if not“good” (—). Pressure observations from HUGO and WYNO and 5-cm soil temperature at GOOD appear as new problems. Daily aggregates of departures from estimated values are listed in the ASI.

Fig. 5.

An excerpt from the automated QA summary report for 21 Sep 1997. The letters in parentheses represent the current sensor status in the qualparm table (S, suspect; W, warning; F, failure), if not“good” (—). Pressure observations from HUGO and WYNO and 5-cm soil temperature at GOOD appear as new problems. Daily aggregates of departures from estimated values are listed in the ASI.

The summary report also includes a listing of station parameters, which have the highest average departure from their estimated values. The ratios of the difference between the observed and estimated values to the standard deviation (Δ, from the spatial test) are averaged over the entire 24-h period. To make these results easier to interpret, the resulting number is scaled to 100, forming an Aggregate Score Index (ASI). An ASI score of 100 means that the average reading departs from its estimated value by twice the standard deviation of the sample. This value can be used to detect a parameter that does not immediately violate Mesonet’s automated QA standards but may be developing a bias that merits attention. ASI scores above 50 generally warrant close attention; those over 100 are usually associated with more serious problems that will likely appear in the flag counts in the top portion of the summary report. The top 10 ASIs are listed in the report in descending order.

The high degree of variability in rainfall patterns poses a special problem for automated quality-assurance techniques. Because localized events may be detected by a single gauge, or areas of enhanced precipitation may fall between gauges, the Mesonet’s automated quality assurance is not permitted to apply flags to rainfall. Instead, the 24-h accumulated precipitation value is compared to an estimated value for the site, using the same techniques as previously described. If a site reports less than 25% of the estimated value, provided that the estimate is at least 12.7 mm (0.5 in.), a report is included in the daily e-mail to the QA manager. Also, if the estimated value for the site is near zero, but the gauge indicates the occurrence of at least two bucket tips, a report is included to the QA manager. In the first instance, a low rainfall report may indicate a clogged rain gauge. In the second instance, water may accumulate and drip slowly through the obstruction over subsequent days, indicating a problem that is otherwise difficult to detect. When rainfall totals appear in the QA report, the QA manager compares them to WSR-88D reflectivity and precipitation estimates to make a subjective decision as to whether the rain gauge needs to be investigated by a technician. Suspect rainfall amounts are listed for a one-month period, making it easier to identify patterns of under- or overreporting rain gauges.

The automated QA is reapplied each day to each of the preceding seven days of data in order to incorporate data that may have been collected after the regular reporting time and to include the QA manager’s revisions to the qualparm table. The automated QA procedure is also repeated on a monthly, three-monthly, and annual basis. This delay allows sufficient time to secure missing data and for additional revisions of qualparm values. For example, a problem detected via a QA analysis method will have the appropriate flag(s) updated in the qualparm table. By reapplying the automated QA after a one-month lag, these updated qualparm values yield more accurate QA flags, which become the permanent QA files.

7. Manual inspection

Subtle problems, such as instrument drift, are sometimes not recognizable through automated analysis. They may be obscured by a small bias in the observations. A thorough visual analysis of Mesonet data via a well-trained meteorologist normally helps identify these subtle problems. In addition, some mesoscale (e.g., drylines entering the network) or localized phenomena (e.g, thunderstorms, heatbursts) may be erroneously flagged by the automated analysis. Thus, manual inspection can also ensure that good data are not improperly flagged.

a. Real-time data monitoring

A number of techniques are used to monitor the quality of the data as they are collected. These include data-collection software, kiosk displays of color-filled contours of various parameters, meteograms, and Web pages.

Data-collection software keeps track of each observation from a station. If a station fails to respond to a request for data, the system queues a request for later collection, a process that is known as “hole collection.” Although this process does not ensure the quality of the actual observations, it does a remarkably effective job at ensuring a complete dataset. The Mesonet maintains an archival rate of 99.9%. Thus, a researcher can be confident that requested data will have been collected and archived.

The QA manager may also monitor incoming Mesonet data during the day. A suite of real-time Web products is typically used to view current data. These include maps of raw Mesonet parameters, maps of derived parameters, 24-h meteogram time-series plots of various Mesonet parameters, and maps of 3-h trends for several parameters. The QA manager may also use real-time kiosks and, of course, raw data files available upon data ingest. Using various Web pages, Mesonet operators also monitor the data as they arrive and report suspicious events to the QA manager.

b. Long-term analysis

Aside from inspection of raw data and various time-series plots, manual inspection also involves contoured maps that represent long-term (several weeks to several months) averages of selected parameters. Contours are determined by simple objective analysis techniques (typically a Barnes analysis), then interpreted through the eyes of the QA manager.

Subtle problems with parameters such as air temperature, dewpoint temperature, and wind speed can best be identified by an analysis of the average conditions at a specific time of day during the period of interest (Fig. 6). Other parameters, such as rainfall and solar radiation, lend themselves well to an analysis of cumulative totals during the period of interest. In either case, subtle problems typically appear as data irregularities that cannot be explained by the QA manager’s meteorological knowledge and experience with local anomalies.

Fig. 6.

Average dew point temperature at 1200 UTC for Aug 1997. The anomaly in southeast Oklahoma is caused by microscale effects. The anomaly in west-central Oklahoma prompted the QA manager to issue a trouble ticket.

Fig. 6.

Average dew point temperature at 1200 UTC for Aug 1997. The anomaly in southeast Oklahoma is caused by microscale effects. The anomaly in west-central Oklahoma prompted the QA manager to issue a trouble ticket.

At a single Mesonet site where similar parameters are measured by several instruments (e.g., wind speed at two levels, or soil temperatures at a number of depths), comparison between instruments can help identify problem sensors. For instance, an objectively analyzed plot of the average difference between two sensors, in association with other maps, can be used to help pinpoint a sensor problem. This method is particularly useful with soil temperature probes because a traditional spatial analysis of soil temperature is limited by the localized nature of soil conditions. A full list of parameters that are evaluated via these visual inspections is given in Table 4.

Table 4.

List of parameters evaluated via long-term visual analyses. All products are monthly averages of the parameter at the indicated times. “Monthly” indicates averages, accumulations, or extrema of all data for that parameter during an entire month.

List of parameters evaluated via long-term visual analyses. All products are monthly averages of the parameter at the indicated times. “Monthly” indicates averages, accumulations, or extrema of all data for that parameter during an entire month.
List of parameters evaluated via long-term visual analyses. All products are monthly averages of the parameter at the indicated times. “Monthly” indicates averages, accumulations, or extrema of all data for that parameter during an entire month.

Allen (1996) recommends several long-term tests to detect problems with data. Some of these tests, such as the maximum relative humidity and examination of spatial consistency of wind speeds, are used routinely in the QA manager’s monthly analyses. Several other techniques, such as comparing actual to estimated solar radiation, are presently under development for inclusion in both automated QA routines and incorporation in monthly analyses. Other longer-term analyses, such as “double mass analysis” techniques, are only now becoming possible as the Mesonet has a long enough historical record for these techniques to be successful.

c. Case-by-case investigation

Not all suspect observations lend themselves to a quick conclusion as to whether or not the data should be flagged. Site-specific characteristics are often discovered where the automated QA and manual inspection indicate potential concern with the data. The QA manager thoroughly investigates these potential problems before sending a technician for a field comparison.

The QA manager’s investigation includes creating time-series graphs depicting data from the site and parameter in question, along with data from neighboring sites. In some cases, when a parameter is highly variable across an area (e.g., soil temperature at 5 cm), the automated spatial test may not properly discern which among several sites has the suspect data. In these situations, flags may be erroneously placed on the wrong station. These techniques enable the QA manager to correctly identify questionable sensors.

The QA manager keeps a daily log of less serious problems so those sites that produce short-lived suspicious data can be monitored for several days before notifying a technician. This strategy is especially useful for determining whether a pyranometer is reporting data spikes due to a loose wire or whether the spikes may be attributable to a bird using the sensor as a temporary perch. Using data from more than one rain event is also preferred when making a determination as to whether a flag is required on a seemingly over- or underreporting rain gauge.

A number of local effects also raise concerns about the quality of meteorological data. For instance, an agricultural field near a station can sometimes affect observations at a Mesonet site. Figure 7 displays the dependence of afternoon temperatures at the Altus site on wind direction. When winds had a northeasterly component, the temperatures at Altus and nearby Tipton were very similar. However, when winds traversed an irrigated farm located south and southwest of the Altus site, the temperature at Altus was as much as 3°C cooler than that observed at Tipton.

Fig. 7.

Dependency of air temperature on wind direction for the Altus and Tipton Mesonet sites during 6–9 Aug 1998.

Fig. 7.

Dependency of air temperature on wind direction for the Altus and Tipton Mesonet sites during 6–9 Aug 1998.

d. Keeping the good data

Daily inspection of the results tabulated by the automated QA analysis also allows the QA manager to track erroneously flagged data. The range test was thought to be least susceptible to producing erroneous QA flags. Nevertheless, the hot, dry summer of 1998 resulted in numerous 5-cm soil temperatures exceeding 50°C. With air temperatures approaching the mid-40°C range, the data were believed to be suspect at worst. After further investigation, the range test threshold for both the 5-cm soil temperatures (under native sod and under bare soil) was increased to 55°C.

Thunderstorms also have caused numerous accurate observations to be flagged erroneously by the automatic QA. Cold-air outflow boundaries and heatbursts oftentimes cause a station to be spatially inconsistent with respect to neighboring sites. Associated wind gusts and pressure dips create data failures during the step test. Visual inspection (such as that illustrated in Fig. 8) allows the QA manager to determine whether such inconsistencies may actually be real events.

Fig. 8.

Pressure spike and corresponding wind gust at the Seiling Mesonet site on 4 Jun 1998.

Fig. 8.

Pressure spike and corresponding wind gust at the Seiling Mesonet site on 4 Jun 1998.

The highly variable nature of rainfall necessitates that rainfall observations flagged as suspicious receive close scrutiny from the QA manager. Figure 9 illustrates how a lone shower can create havoc for an automated spatial QA routine. The 1-mm rainfall total observed at Broken Bow (southeast corner of Oklahoma) appeared as an overestimate because all neighboring sites reported no rain. By comparing with WSR-88D data for the area, the QA Manager determined that this report was valid.

Fig. 9.

A comparison of Mesonet rainfall reports with WSR-88D data for 14 Jul 1998. (a) Rainfall totals (in mm) for southeast Oklahoma for a 24-h period ending at 0000 UTC 15 Jul 1998. (b) WSR-88D base reflectivity from KSRX (Fort Smith, AR) on 14 Jul 1998 at 0044 UTC. The QA manager used the WSR-88D image to confirm that the report of precipitation at Broken Bow was accurate.

Fig. 9.

A comparison of Mesonet rainfall reports with WSR-88D data for 14 Jul 1998. (a) Rainfall totals (in mm) for southeast Oklahoma for a 24-h period ending at 0000 UTC 15 Jul 1998. (b) WSR-88D base reflectivity from KSRX (Fort Smith, AR) on 14 Jul 1998 at 0044 UTC. The QA manager used the WSR-88D image to confirm that the report of precipitation at Broken Bow was accurate.

Logs of data failures detected by the automated QA system provide additional sources of feedback to the QA manager. First, one is able to identify situations in which QA test thresholds may be too stringent, such as with the soil temperatures. Second, such logs may indicate that the automated QA routines do not perform well under certain weather scenarios and need to be modified. Third, the QA manager may recognize a localized or unique weather event that was erroneously flagged by the automated QA. The qualparm table may be used to downgrade the severity of flags determined by the automated QA processes for the duration of the event (flag 5 or 6; see Table 2). For example, a thunderstorm outflow event may be indicated in the qualparm table such that the results of the other QA tests will be ignored and the final flag for the corresponding data will be reset to zero. The ability to downgrade QA flags was not yet operational at the time of this writing.

8. Communication within the QA system

Evolution of the Mesonet QA system includes a refinement of communication between all components of the QA system (Arndt et al. 1998). More efficient communication yields more accurate data flagging and a quicker response by Mesonet personnel to developing or existing data quality problems.

The QA manager is responsible for coordinating incoming QA information from a variety of sources (e.g., automated reports, visual inspection, technician input, data user reports), for determining the reality of an apparent problem, and for issuing a call-to-action in the form of a trouble ticket. Use of the Internet, cellular telephones, and paging technology speeds the response time to developing problems.

To keep Mesonet personnel informed about recent and current QA-related events, the QA manager prepares a monthly report, which summarizes instrument transactions and documents a subjective assessment of the network’s performance (Fig. 10). The report provides a synopsis of QA concerns and actions during the previous month, and notes the overall status of data quality from the Mesonet. Significant observations of meteorological events are included in the discussion. The report is sent to interested parties via e-mail and a paper copy is retained for future reference.

Fig. 10.

An excerpt from the QA manager’s monthly report for Feb 1998. The QA manager includes a brief synopsis of the performance of the network followed by the status of trouble tickets, sorted by parameter.

Fig. 10.

An excerpt from the QA manager’s monthly report for Feb 1998. The QA manager includes a brief synopsis of the performance of the network followed by the status of trouble tickets, sorted by parameter.

a. The trouble ticket

The trouble ticket is the fundamental method used to report and record sensor problems, and to ensure and log their resolution. When a data problem is identified and determined to be legitimate, a trouble ticket is issued. The outgoing ticket contains information (station, parameter, description of problem, date/time of problem onset, urgency), which assists the Mesonet technician with assessment of the problem. The technician works the trouble ticket into the maintenance agenda based on the problem’s urgency, solvability, and the availability of replacement sensors.

Using information from the trouble ticket and on-site analysis, the technician takes action to resolve the problem. The sensor may be serviced or replaced, or the technician may discover that the data problem is not attributable to the sensor. The trouble ticket carries the technician’s in-field observations and decisions (method of fix, fix date/time, instrument transaction) to the QA manager. If necessary, the QA manager updates metadata files (containing information regarding instrument transactions and location ledgers) and the qualparm table. The trouble ticket is archived (electronically and to a paper file) because it contains valuable historical information (metadata) about a data problem from a specific sensor.

b. Internet communication in the QA system

When the QA manager enters the problem description for a trouble ticket, there is an option to send an electronic form of the trouble ticket via e-mail to the appropriate personnel. An electronic trouble ticket serves two purposes: 1) it alerts technicians who are based away from the central facility and 2) maintenance actions can begin before a paper ticket reaches the technician.

Three World Wide Web (WWW) templates are available to Mesonet personnel. The first allows a Mesonet employee to report apparent problems to the QA manager and the technician with maintenance responsibilities at the affected station. Each category listed on a trouble ticket is duplicated on the WWW page (Fig. 11a). When the completed template is submitted, an e-mail message is sent to the QA manager, giving notice of the potential problem. The QA manager then decides whether to issue a formal trouble ticket.

Fig. 11.

WWW pages used to streamline the communication of trouble tickets and instrument decisions: (a) trouble ticket submission page and (b) trouble ticket resolution submission page.

Fig. 11.

WWW pages used to streamline the communication of trouble tickets and instrument decisions: (a) trouble ticket submission page and (b) trouble ticket resolution submission page.

Another template is used by technicians to report information stemming from the resolution of a trouble ticket (Fig. 11b). Through this template, a technician may relay the results of a station visit to the QA manager, allowing the QA manager to adjust qualparm settings to appropriate values long before the paper trouble ticket is processed.

A third template involves sensors that require instrument-specific calibration coefficients (rain gauges and pyranometers; not shown). When these instruments are relocated, calibration tables must be updated quickly so the proper coefficients can be immediately applied to the outgoing data. The WWW interface produces a formatted e-mail message, which details the instrument change.

An instrument “residence-time” database is another tool available via the WWW to assist Mesonet personnel (Fig. 12). A plot shows the time, in months, that each sensor of a given type has been on location. The QA Manager often considers residence time when making decisions regarding questionable sensors and symptoms of instrument “drift.” In addition, Mesonet technicians use these maps to plan site visitation itineraries, which improves the rotation schedule of Mesonet sensors.

Fig. 12.

Residence time (in months) of Mesonet barometers. The values in the Texas panhandle represent two three-station clusters that are too closely spaced to be displayed at their proper geographic location.

Fig. 12.

Residence time (in months) of Mesonet barometers. The values in the Texas panhandle represent two three-station clusters that are too closely spaced to be displayed at their proper geographic location.

Similarly, a list of unresolved trouble tickets is kept online for review by Mesonet personnel. This file helps technicians integrate lower-priority trouble tickets (which do not require an urgent trip) into their visitation schedules.

c. The use of cellular and paging technology

Oftentimes, new concerns about data quality arise while Mesonet technicians are away from the central facility or are otherwise unable to access an Internet connection. The use of cellular and alphanumeric paging technology maintains a reliable line of communication with these technicians. Frequently, a Mesonet operator will submit one of the WWW templates for a technician based on details given in a cellular phone conversation. Alphanumeric pagers are also used to send urgent messages to Mesonet personnel.

Each morning, an operator dispatches a brief report to the technicians via an alphanumeric pager. This report contains information about the status of observed problems with data from particular stations and includes documentation of any major power failures or communication breakdowns within the network during the previous 24 h. Technicians and the QA manager also periodically receive observations of severe wind speeds, which may cause damage at a Mesonet site, via their alphanumeric pagers.

9. Five years of quality-assurance experiences

As with any operational network, some factors cause frequent problems within the system. Lightning, spiders, insects, grass fires, gophers, and vandalism are several examples of the nemeses with which Mesonet personnel must contend. Experience over five years has shown that remote weather stations provide ideal habitats for a number of insects and animals. Spiders like to nest in the rain gauge funnels and insects find that pressure tubes provide nice housing. Burrowing animals chew through cables connected to the soil temperature probes, and, on occasion, have even been found to remove the probes themselves. Sadly, humans also have plagued a few of our sites, with vandalism and theft of equipment. Some of the notable events causing problems with the Mesonet include

  • a horse using a temperature sensor radiation shield as a scratching post;

  • tumbleweeds accumulating within the Mesonet enclosure, insulating soil temperatures;

  • grass fires melting instruments and housings (but providing interesting data during the event);

  • airborne sand and dust during a drought clogging rain gauges; and

  • cattle getting inside the Mesonet enclosure, disturbing soil probes and knocking over a solar radiation tripod.

Quality assurance also has identified subtle effects that result from changes in the environment, and not a fault with the instruments. Soil temperature biases in monthly analyses have led technicians to discover problems with erosion and sterilant leaching from the bare soil plots. Technicians now routinely check the depth of the soil temperature sensors when they visit a site.

Another example of how the QA process feeds back into network design occurred in March 1995 when six Mesonet stations in southwest Oklahoma simultaneously reported barometric pressures that the QA manager suspected were too low. This anomaly was detected via visual inspection of a plot containing a 7-day average of sea level pressure at 1800 UTC. The simultaneous but slight underreporting by Mesonet barometers in close proximity to each other revealed an unexpected design condition. Rain had frozen over the barometer port, sealing it from the outside atmosphere. As the air cooled inside the sealed box, the pressure decreased. Project staff decided that because this event represented a rare occurrence, a technical solution could introduce more serious problems; thus no alterations were made to the station design. This decision was, in part, based upon an earlier experience, in which a porous filter had been added to the barometer ports to prevent debris and insects from nesting in the tubes. The filters tended to absorb water and temporarily clog the tubes. Shortly after their addition to the network, the filters had to be removed.

Results from QA processes, combined with human experience and meteorological knowledge, can lead to the discovery of previously unknown microscale phenomena. One example is the relatively frequent indication of data problems, as detected by the automated spatial analysis routine, at the Medicine Park station in southwest Oklahoma during overnight periods. The station is located within an isolated mountain range, bordered by flat grasslands. Quite often, this station, because of its elevation, extends through the base of a strong temperature inversion, which otherwise goes unobserved at nearby Mesonet sites. During these events, the higher temperatures and lower moisture values at the Medicine Park site appear to be anomalous but in fact they are real. A consistency between temperature data from 1.5 and 9.0 m, as indicated by the like-instrument comparison test, is an effective method of identifying such “inversion pokers.”

Anomalies in wind direction at sites along the Arkansas River in eastern Oklahoma (Sallisaw and Webbers Falls; Fig. 1) caught the attention of Mesonet staff. When winds were light across the region, these sites developed an east wind anomaly. Initially, Mesonet staff suspected a problem with the alignment of the wind sensors, but upon closer investigation, they discovered that the anomaly was real. A similar, but weaker, anomaly was discovered along the Red River in southeast Oklahoma.

The Ouachita Mountains in southeast Oklahoma have a pronounced effect on mesoscale weather patterns; they are large enough to sustain a mountain/valley circulation, as detected by the Mesonet site at Wister. Boundary layer flow during the afternoon is uniform with surrounding sites, but in the absence of large-scale forcing, the wind at night becomes light northerly. Obstacle flow also can be seen in the neighboring counties of Leflore, Pushmataha, and McCurtain.

During 1998, Mesonet personnel discovered that heatbursts are a relatively common event. Heatbursts often affect only one or two stations, and appear as anomalously warm and dry conditions at night. During the 5-year period of 1994–98, more than 50 of these “rare” events were detected within Oklahoma (e.g., see MacKeen et al. 1998). Heatbursts invariably appear as data problems in the automated QA, but human judgment is able to override these indications and preserve a record of these events for future study.

10. Conclusions

The Oklahoma Mesonet’s QA system represents a compilation of processes that range from simple testing to sophisticated analysis. Automated processes prompt human attention, while human experience shapes the design and interpretation of the automated processes. The system utilizes multiple techniques of data and instrument analysis. When applied individually, these components have unique strengths and weaknesses. When a concerted effort is made to compile their results and integrate that information via efficient communication, a comprehensive QA system emerges that accurately assesses the quality of both past and present data from the Oklahoma Mesonet.

Constant improvement of the QA system is necessary to better realize its goals of high quality data in both real time and in its archives. Because the Mesonet uses a complex QA scheme of instrument calibration, field comparisons, automated routines, and manual inspection, it is possible to use feedback from one component of the system to improve another component. For example, knowledge gained from postcalibration of sensors helps to optimize the desired residence time of certain instruments so that data quality is maximized and unnecessary sensor rotation is minimized. Postcalibration of sensors has proven to be beneficial when determining preinstallation calibration strategies. On-site instrument intercomparisons help validate or refute the QA manager’s visual interpretations. This feedback, in turn, is important to the continual refinement of the automated QA routines.

It is important that the QA system be scrutinized as rigorously as the data to ensure continual improvements in the data quality of the archives and to achieve the optimum quality of real-time data. Thoughtful adjustments and additions to the QA system are sought to improve the accuracy and response time of the QA system. Ideally, improvements in the QA system should take advantage of the latest available technology and techniques in data analysis. However, it should be remembered that the purpose of these QA flags is to provide guidance to those who use data from the Oklahoma Mesonet. Because no data are altered, the user can make the final judgment as to the quality of the data.

The role of the experienced human is essential to the QA process. Many actions and decisions rely upon the knowledge and experience of the technicians, laboratory personnel, the QA manager, and external users. Human experience is critical when assessing the reality of a questionable event. Human interaction with the Mesonet QA system has resulted in the discovery of many subtle, but real, meteorological events, and has provided a capability to distinguish these phenomena from data problems.

Acknowledgments

The number of authors on this paper is an indication of the complexity of the QA process. Many people have contributed to producing quality data from the Oklahoma Mesonet. In particular, we would like to acknowledge Dr. Fred Brock, architect of the Mesonet QA procedures; Dale Morris, Putnam Reiter, Jared Bostic, and the Mesonet operators; our laboratory and field technicians—Ken Meyers, Gary Reimer, Bill Wyatt, David Grimsley, James Kilby, and Leslie Cain—and last but not least, our former QA managers: David Shellberg, Curtis Marshall, Jeff Basara, and JerryBrotzge. We also appreciate the guidance provided by the Mesonet Steering Committee: Ken Crawford, Ron Elliott, Steve Stadler, Howard Johnson, Mike Eilts, Al Sutherland, and former members Chuck Doswell, Gerritt Cuperus, and Jim Duthie. The Oklahoma Mesonet was financed by funds from the Exxon Oil Overcharge Settlement Fund as administered by the Oklahoma Department of Commerce. We gratefully acknowledge the State of Oklahoma’s continuing financial contribution to annual operational costs of the Mesonet.

REFERENCES

REFERENCES
Allen, R. G., 1996: Assessing integrity of weather data for reference evapotranspiration estimation. J. Irrig. Drain. Eng., 122 (2), 97–106.
Arndt, D. S., M. A. Shafer, S. E. Fredrickson, and J. P. Bostic, 1998:Quality assurance at the Oklahoma Mesonet: A systems viewpoint. Preprints, 10th Symp. on Meteorological Observations and Instrumentation, Phoenix, AZ, Amer. Meteor. Soc., 349–354.
Barnes, S. L., 1964: A technique for maximizing details in numerical weather map analysis. J. Appl. Meteor., 3, 396–409.
Brock, F. V., K. C. Crawford, R. L. Elliott, G. W. Cuperus, S. J. Stadler, H. L. Johnson, and M. D. Eilts, 1995: The Oklahoma Mesonet, a technical overview. J. Atmos. Oceanic Technol., 12, 5–19.
Brotzge, J. A., and K. C. Crawford, 2000: Estimating sensible heat fluxes from the Oklahoma Mesonet. J. Appl. Meteor., 39, 102–116.
Crawford, K. C., D. S. Arndt, D. J. Shellberg, and B. M. Brus, 1995:A comparison of differences in meteorological measurements made by the Oklahoma Mesonetwork at two co-located ASOS sites. Preprints, 11th Int. Conf. on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, Dallas, TX, Amer. Meteor. Soc., 299–303.
Elliott, R. L., F. V. Brock, M. L. Stone, and S. L. Harp, 1994: Configuration decisions for an automated weather station network. Appl. Eng. Agric., 10 (1), 45–51.
Fiebrich, C. A., and K. C. Crawford 1998: An investigation of significant low-level temperature inversions as measured by the Oklahoma Mesonet. Preprints, 10th Symp. on Meteorological Observations and Instrumentation, Phoenix, AZ, Amer. Meteor. Soc., 337–342.
Fredrickson, S. E., J. A. Brotzge, D. Grimsley, and F. V. Brock, 1998:An unusual Oklahoma Mesonet temperature sensor problem (nearly) slips by in-situ, real-time, short-term and long-term QA efforts . . . and why this could happen to you. Preprints, 10th Symp. on Meteorological Observations and Instrumentation, Phoenix, AZ, Amer. Meteor. Soc., 343–348.
Gandin, L. S., 1988: Complex quality control of meteorological observations. Mon. Wea. Rev., 116, 1137–1156.
Gill, G. C., 1979: Development of a small radiation shield for air temperature measurements on drifting buoys. NOAA Data Buoy Office Contract 01-7-038-827(11). [Available from NOAA Data Buoy Office, Bay St. Louis, MS 39520.]
.
MacKeen, P., D. L. Andra, and D. A. Morris, 1998: The 22–23 May 1996 heatburst: A severe wind event. Preprints, 19th Conf. on Severe Local Storms, Minneapolis, MN, Amer. Meteor. Soc., 510–513.
Meek, D. W., and J. L. Hatfield, 1994: Data quality checking for single station meteorological databases. Agric. Forest Meteor., 69, 85–109.
Meyer, S. J., and K. G. Hubbard, 1992: Nonfederal automated weather stations and networks in the United States and Canada: A preliminary survey. Bull. Amer. Meteor. Soc., 73, 449–457.
Richardson, S. J., 1995: Automated temperature and relative humidity calibrations for the Oklahoma Mesonetwork. J. Atmos. Oceanic Technol., 12, 951–959.
——, F. V. Brock, S. R. Semmer, and C. Jirak, 1999: Minimizing errors associated with multiplate radiation shields. J. Atmos. Oceanic Technol., 16, 1862–1872.
Shafer, M. A., T. Hughes, and J. D. Carlson, 1993: The Oklahoma Mesonet: Site selection and layout. Preprints, Eighth Symp. on Meteorological Observations and Instrumentation, Anaheim, CA, Amer. Meteor. Soc., 231–236.
Snyder, R. L., and W. O. Pruitt, 1992: Evapotranspiration data management in California. Irrigation and Drainage, Saving a Threatened Resource—In Search of Solutions, T. E. Engman, Ed., American Society of Civil Engineers, 128–133.
Stanhill, G., 1992: Accuracy of global radiation measurements at unattended, automatic weather stations. Agric. Forest Meteor., 61, 151–156.
Wade, C. G., 1987: A quality control program for surface mesometeorological data. J. Atmos. Oceanic Technol., 4, 435–453.

APPENDIX

Data Quality Subcommittees Established by the Mesonet Steering Committee

Site Standards

Recommend technical standards by which potential sites are evaluated.

Membership:

  • Claude Duchon (Chair), University of Oklahoma, Norman

  • Ron Hanson, USGS, Oklahoma City

  • John F. Stone, Oklahoma State University, Stillwater

  • Steve Stadler, Oklahoma State University, Stillwater

Site Selection

Recommend appropriate specific sites and alternative sites, based on recommendations from the Site Standards Committee.

Membership:

  • Carroll Scoggins (Chair), Army Corps of Engineers, Tulsa

  • Les Showell, National Severe Storms Laboratory, Norman

  • John Lambert, National Weather Service, Norman

  • John Damicone, Oklahoma State University, Stillwater

  • Ron Elliott, Oklahoma State University, Stillwater

Parameter Selection

Recommend a prioritized list of parameters to be measured at each site consistent with overall project goals and resources.

Membership:

  • Ron Elliott (Chair), Oklahoma State University, Stillwater

  • Ron Alberty, WSR-88D Operational Support Facility, Norman

  • Dennis McCarthy, National Weather Service Forecast Office, Norman

  • Kathy Peter, USGS, Oklahoma City

Sensor Specification

Using input from the Parameter Selection Subcommittee, recommend the standards for sensor accuracy and reliability.

Membership:

  • Marv Stone (Chair), Oklahoma State University, Stillwater

  • Sam Harp, Oklahoma State University, Stillwater

  • Ken Brown, National Weather Service Forecast Office, Norman

  • Fred Brock, University of Oklahoma, Norman

Station Maintenance

Recommend the standards and strategy for meeting station maintenance requirements.

Membership:

  • Sam Harp (Chair), Oklahoma State University, Stillwater

  • Jerry Hunt, National Weather Service Forecast Office, Norman

  • Sherman Fredrickson, National Severe Storms Laboratory, Norman

Quality Assurance

Recommend the activities necessary to maintain data quality.

Membership:

  • Fred Brock (Chair), University of Oklahoma, Norman

  • Bill Bumgarner, WSR-88D Operational Support Facility, Norman

  • Sherman Fredrickson, National Severe Storms Laboratory, Norman

  • Claude Duchon, University of Oklahoma, Norman

Data Management

Develop and recommend data manipulation strategies to ensure quality data are available on demand in an efficient manner and format.

Membership:

  • Howard Johnson (Chair), Oklahoma Climatological Survey, Norman

  • W. S. Fargo, Oklahoma State University, Stillwater

  • Steve Stadler, Oklahoma State University, Stillwater

  • Wes Roberts, University of Oklahoma, Norman

Footnotes

* Additional affiliation: Oklahoma Climatological Survey, Mesonet Project, Norman, Oklahoma.

Corresponding author address: Mark A. Shafer, Oklahoma Climatological Survey, 100 E. Boyd St., Suite 1210, Norman, OK 73019.