Data Quality Control Based on Self-Consistency

Reinhold Steinacker Department of Meteorology and Geophysics, University of Vienna, Vienna, Austria

Search for other papers by Reinhold Steinacker in
Current site
Google Scholar
PubMed
Close
,
Dieter Mayer Department of Meteorology and Geophysics, University of Vienna, Vienna, Austria

Search for other papers by Dieter Mayer in
Current site
Google Scholar
PubMed
Close
, and
Andrea Steiner Department of Meteorology and Geophysics, University of Vienna, Vienna, Austria

Search for other papers by Andrea Steiner in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Conducting meteorological measurements, one is always confronted with a wide variety of different types of errors and with the decision of how to correct data for further use, if necessary. The selection of an adequate quality control (QC) procedure out of a wide range of methodologies depends on the properties of the observed parameter such as spatial or temporal consistency. But the intended data application (e.g., model-independent data analysis) or the availability of prior knowledge also has to be taken into account. The herein-presented self-consistent and model-independent QC process makes use of the spatial and temporal consistency of meteorological parameters. It is applicable to measurements featuring a high degree of autocorrelation with regard to the resolution of the observational network in space and time. The presented QC procedure can mathematically be expressed as an optimization problem minimizing the curvature of the analyzed field. This results in a matrix equation that can be solved without needing to converge iterations. Based on the resulting deviations and, if applied, on their impacts on the cost function, station values are accepted, corrected, or identified as outliers and hence dismissed. Furthermore, it is pointed out that this method is able to handle complicated station distributions, such as clustered stations or inhomogeneous station densities. This QC method is not only an appropriate tool for case studies but also for model validation and has been proving itself as a preprocessing tool for operational meso- and micrometeorological analyses.

Corresponding author address: Reinhold Steinacker, Dept. of Meteorology and Geophysics, Althanstrasse 14, 1090 Vienna, Austria. E-mail: reinhold.steinacker@univie.ac.at

Abstract

Conducting meteorological measurements, one is always confronted with a wide variety of different types of errors and with the decision of how to correct data for further use, if necessary. The selection of an adequate quality control (QC) procedure out of a wide range of methodologies depends on the properties of the observed parameter such as spatial or temporal consistency. But the intended data application (e.g., model-independent data analysis) or the availability of prior knowledge also has to be taken into account. The herein-presented self-consistent and model-independent QC process makes use of the spatial and temporal consistency of meteorological parameters. It is applicable to measurements featuring a high degree of autocorrelation with regard to the resolution of the observational network in space and time. The presented QC procedure can mathematically be expressed as an optimization problem minimizing the curvature of the analyzed field. This results in a matrix equation that can be solved without needing to converge iterations. Based on the resulting deviations and, if applied, on their impacts on the cost function, station values are accepted, corrected, or identified as outliers and hence dismissed. Furthermore, it is pointed out that this method is able to handle complicated station distributions, such as clustered stations or inhomogeneous station densities. This QC method is not only an appropriate tool for case studies but also for model validation and has been proving itself as a preprocessing tool for operational meso- and micrometeorological analyses.

Corresponding author address: Reinhold Steinacker, Dept. of Meteorology and Geophysics, Althanstrasse 14, 1090 Vienna, Austria. E-mail: reinhold.steinacker@univie.ac.at
Save
  • Andersson, E., and H. Järvinen, 1999: Variational quality control. Quart. J. Roy. Meteor. Soc., 125, 697–722.

  • Baker, N. L., 1992: Quality control for the navy operational atmospheric database. Wea. Forecasting, 7, 250–261.

  • Barber, C. B., D. P. Dobkin, and H. Huhdanpaa, 1996: The Quickhull algorithm for convex hulls. ACM Trans. Math. Software, 22, 469–483.

    • Search Google Scholar
    • Export Citation
  • Barnes, S. L., 1964: A technique for maximizing details in numerical weather map analysis. J. Appl. Meteor., 3, 396–409.

  • Feng, S., Q. Hu, and W. Qian, 2004: Quality control of daily meteorological data in China, 1951–2000: A new dataset. Int. J. Climatol., 24, 853–870.

    • Search Google Scholar
    • Export Citation
  • Fiebrich, C. A., and K. C. Crawford, 2001: The impact of unique meteorological phenomena detected by the Oklahoma mesonet and ARS micronet on automated quality control. Bull. Amer. Meteor. Soc., 82, 2173–2187.

    • Search Google Scholar
    • Export Citation
  • Gandin, L. S., 1988: Complex quality control of meteorological observations. Mon. Wea. Rev., 116, 1137–1156.

  • Hubbard, K. G., and J. You, 2005: Sensitivity analysis of quality assurance using the spatial regression approach—A case study of the maximum/minimum air temperature. J. Atmos. Oceanic Technol., 22, 1520–1530.

    • Search Google Scholar
    • Export Citation
  • Ingleby, N. B., and A. C. Lorenc, 1993: Bayesian quality control using multivariate normal distributions. Quart. J. Roy. Meteor. Soc., 119, 1195–1225.

    • Search Google Scholar
    • Export Citation
  • Jolliffe, I., and D. Stephenson, 2003: Forecast Verification: A Practitioner’s Guide in Atmospheric Science. John Wiley and Sons, 240 pp.

    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., 1981: A global three-dimensional multivariate statistical interpolation scheme. Mon. Wea. Rev., 109, 701–721.

  • Lorenc, A. C., and O. Hammon, 1988: Objective quality control of observations using Bayesian methods: Theory, and a practical implementation. Quart. J. Roy. Meteor. Soc., 114, 515–543.

    • Search Google Scholar
    • Export Citation
  • Pöttschacher, W., R. Steinacker, and M. Dorninger, 1996: VERA - a high resolution analysis scheme for the atmosphere over complex terrain. MAP Newsletter, Vol. 5, Mesoscale Alpine Programme Office, Zurich, Switzerland, 64–65.

    • Search Google Scholar
    • Export Citation
  • Reek, T., S. R. Doty, and T. W. Owen, 1992: A deterministic approach to the validation of historical daily temperature and precipitation data from the cooperative network. Bull. Amer. Meteor. Soc., 73, 753–762.

    • Search Google Scholar
    • Export Citation
  • Shafer, M. A., C. A. Fiebrich, D. S. Arndt, S. E. Fredrickson, and T. W. Hughes, 2000: Quality assurance procedures in the Oklahoma mesonetwork. J. Atmos. Oceanic Technol., 17, 474–494.

    • Search Google Scholar
    • Export Citation
  • Steinacker, R., C. Häberli, and W. Pöttschacher, 2000: A transparent method for the analysis and quality evaluation of irregularly distributed and noisy observational data. Mon. Wea. Rev., 128, 2303–2316.

    • Search Google Scholar
    • Export Citation
  • Steinacker, R., and Coauthors, 2006: A mesoscale data analysis and downscaling method over complex terrain. Mon. Wea. Rev., 134, 2758–2771.

    • Search Google Scholar
    • Export Citation
  • Wade, C. G., 1987: A quality control program for surface mesometeorological data. J. Atmos. Oceanic Technol., 4, 435–453.

  • Wilks, D., 1995: Statistical Methods in the Atmospheric Sciences. Academic Press, 467 pp.

  • WMO, 2008: Guide to Meteorological Instruments and Methods of Observation. 7th ed. WMO-8, World Meteorological Organization, Geneva, Switzerland, 681 pp.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 4421 3063 142
PDF Downloads 651 113 10