1. Introduction

Climate change has become an important issue, because increasing evidence suggests consistent warming trends over the past century, with a faster warming rate over land compared to oceans (Houghton et al. 2001). More and more efforts have been devoted to the assessment of climate change and their impacts. However, one needs long-term homogeneous records of climate data to characterize climate variability and climate change in the past, and to validate numerical model simulations. It is imperative to conduct quality assurance and homogenization of climate data before these data are used for various climate studies, especially climate change–related studies.

Atmospheric circulation plays an essential role in the climate system because of its effects on the distribution of heat and moisture over the globe. Surface atmospheric pressure is an important variable that describes atmospheric circulation. Variations in surface pressure should also reflect variations in surface temperature, because the two variables are related to each other thermodynamically. Therefore, analysis of surface atmospheric pressure is critical to our understanding of climate variability and climate change.

Several studies on the collection and analysis of atmospheric pressure data have been carried out lately. As a result, several good quality pressure datasets of global or regional coverage have been developed, mainly to provide vital inputs for numerical model studies of global climatic variations and changes (e.g., Smith and Reynolds 2003; Kaplan et al. 2000; Allan et al. 1996; Trenberth and Paolino 1980). Many data quality–related problems were found and corrected in these studies. These problems include data errors and discontinuities or inhomogeneities, and high-latitude station data problems (which are reported to have arisen from a lack of data availability for the Arctic region).

In the mean time, there have been several studies using Canadian pressure data. Slonosky and Graham (2005) developed a Canadian monthly mean station pressure (SP) dataset with 71 stations that have data records for 50–130 yr. They found strong correlations between the variability of atmosphere circulation and surface temperature anomalies. They also reported several major inhomogeneities in the dataset. Nkemdirim and Budikova (2001) examined trends in monthly mean sea level pressure (SLP) in western Canada using data from 51 stations for the period from 1956 to 1993, and reported a significant decline in annual mean and winter mean SLP over the Arctic.

However, the original records of surface atmospheric pressure are hourly measurements, from which the commonly used monthly or daily mean pressure values are derived. Unfortunately, the hourly pressure data archived in Environment Canada (EC) have not undergone a quality control (QC) or quality assurance (QA) procedure (except at times for which missing data are flagged). Slonosky and Graham (2005) corrected some problems in their analysis of monthly pressure data (although their corrections are not physically based and are applied to monthly data), while Nkemdirim and Budikova (2001) did not (and hence their results are most likely unreliable). A high-quality homogeneous pressure database is essential for various climate studies; hourly pressure data of high quality are particularly valuable for studying extremes such as atmospheric storminess. Therefore, the goal of the current study is to develop a comprehensive quality assurance system for hourly pressure data.

The necessity of applying a QA procedure to meteorological data has long been recognized. The earliest QA systems were developed for radiosonde data (Gandin 1988; Collins and Gandin 1990). However, more and more effort has been directed toward developing QA systems for high-temporal-resolution surface meteorological data, such as daily or hourly data (Kunkel et al. 1998; Graybeal et al. 2004; Shafer et al. 2000). A complex QA procedure consists of a series of checks on data, with the results obtained from these checks being used systematically to determine whether or not a value is suspicious and how to correct the suspicious value, if possible. Because not all flagged data are erroneous, a complex QA procedure should check all flagged data to screen out those most suspicious values (for correction or exclusion) and to remove flags from data that are deemed consistent with other reliable data. This procedure is usually called the decision-making method (DMM) (Gandin 1988; Graybeal et al. 2004). A modern complex QA system is used not only to identify but also to correct suspicious data whenever possible.

The EC digital archive contains pressure data from 1953 to date. For the early decades, data were digitized from original paper forms, without any quality control performed after digitization. Even for the real-time data (those from electronic reports), the QC procedure is quite limited according to the EC National Archive hourly data quality control documents published on the EC’s Web site (Environment Canada 2004). Thus, a QA procedure for hourly pressure data is developed in this study with the goal of combining existing techniques with a statistical homogeneity test and fitting them to Canadian historical data.

In this study, we develop a QA procedure for Canadian hourly pressure data. The data, QA procedure, and homogeneity test used are described in sections 2, 3, and 4, respectively. Section 5 describes the error correction algorithms. The corrected data series are analyzed in section 6, with some concluding remarks in section 7.

2. Data

Surface atmospheric pressure is usually recorded for both the station elevation and mean sea level. Generally, atmospheric pressure values at the station elevation are called SP and are calculated from the station barometer readings. Mean SLP is derived from the SP, so that the barometric pressures for stations with different elevations can be compared at a common level (mean sea level) for synoptic purposes. Generally, SP data should be more reliable than SLP because fewer calculations are involved. However, SLP data have been used quite often for various purposes, such as constructing atmospheric circulation indicators (e.g., Wright 1984; Jones et al. 1999), developing long-range climate forecast models (e.g., Christensen and Eilbert 1985), and analyzing severe weather phenomena (e.g., Wang et al. 2006; Alexander et al. 2005). Therefore, high-quality data for both the station elevation and mean sea level are needed for various studies.

In this study we apply the QA system to as many stations as possible to support our interest in producing a gridded pressure dataset in the near future and in using the quality data for future generations of global reanalysis, such as the twentieth-century reanalysis project (Compo et al. 2006). There are 1085 stations available for both SP and SLP data in the EC data archive. Only stations with continuous records of at least 1 yr and at least eight reports per day were included in this study. (Although at most stations atmospheric pressure is reported hourly, with 24 measurements per day, some stations either have only one report every 3 or 6 h or have hourly reports for only part of day, e.g., from 0300 to 1600 UTC. The number of pressure reports per day could vary from station to station and/or from one period to another.) Because SLP data are derived from SP data, and we will use both elements for QA, the checking procedure will be applied to data only when both SP and SLP data are available. A total of 761 stations (see Fig. 1) are analyzed in the study.

3. The quality assurance system

The QA system proposed here consists of five components. These include checking for upper and lower climatological thresholds/limits, temporal pressure changes, and hydrostatic, temporal, and internal consistencies. For each station, all valid (nonmissing) values are subject to these five checks. Based on the results of these checks, a decision regarding either acceptance, correction, or rejection of the data is made.

d. Hydrostatic check (HC)

The hydrostatic check has been used routinely in upper-air radiosonde data quality control (Gandin 1988; Collins and Gandin 1990). It plays a crucial role in identifying errors of either height, pressure, or temperature at mandatory isobaric surfaces. In this study, we use it alone to detect and correct random errors in both station and mean sea level pressure data, and we also combine it with a statistical homogeneity test to detect and correct systematic errors (see section 4 below).

For station pressure P_z and sea level pressure P₀, the hydrostatic check is based on the hydrostatic model {Saucier 1955, his Eq. [3.07(1)]}

View Expanded

where Z is the station elevation (m), R is the gas constant for dry air, T₀ = 273.15 K, g is the acceleration of gravity, a is the standard lapse rate (0.0065°C m⁻¹), and T_dry is the average of the current dry-bulb temperature and the dry-bulb temperature recorded 12 h earlier (°C).

However, since November 1976, the following formula is used in Canada for calculation of mean sea level pressure (Savdie 1982; WMO 1954):

View Expanded

where

View Expanded

and

View Expanded

The T_mv is the mean virtual temperature of the fictitious air column between the height of the station and the mean sea level. The third term in (2b) represents a humidity correction, where e_s is the surface vapor pressure and C_h(Z) is the humidity correction factor (a function of Z). The last term F(T_dry) accounts for correction of plateau effects (see Savdie 1982), and b₁, b₂, and b₃ are plateau correction parameters specific to each station. Note that neglecting humidity and plateau correction, the combination of Eqs. (2a) and (2b) is equivalent to (1).

The hydrostatic residuals R_z are defined as

View Expanded

where Z is the recorded current station elevation, which is taken from Environment Canada’s station information system (SIS) [i.e., a metadata database that also includes historical station information, such as previous station location(s) (latitude–longitude), elevation(s), etc.] and can be assumed correct with good confidence (because the current station elevations in our SIS are very accurate), and Z_m is the estimation of the station elevation obtained by substituting the related hourly P₀, P_z, or T_dry values in model (1). In the absence of data error(s), R_z values shall be very close to zero. A tolerance of R_z is used here to allow for small errors in the value of either P₀ or P_z, or even in the dry-bulb temperature T_dry or the recorded station elevation Z (but undocumented large elevation changes can still be identified by this check).

To be more confident about the results of a hydrostatic check that uses dry-bulb temperatures, the limits check is also performed on dry-bulb temperature [which should range between −55° and 40°C, according to the EC hourly data QC document; see Environment Canada (2004)]. The results show that outliers are found only for 34 out of 761 stations, and the outlier rate is fairly low in general. We flag the R_z values that are associated with outlier(s) of dry-bulb temperature for further analysis. As an additional measure, we also tested the sensitivity of the estimated station elevation in (1) to errors in pressure and temperature records separately, and found that the estimated station elevation is more sensitive to pressure errors than to temperature errors. For example, for a station at 400-m elevation, an error of 1 hPa in the pressure data (station- or sea level pressure) will result in a difference of 8.5 m in the estimated elevation Z_m, while an error of 1°C in the temperature data will only result in a difference of 1.5 m in Z_m. Therefore, in this study, we assume that the recorded hourly dry-bulb temperature values are correct in general, but carefully analyze those values associated with outlier temperature(s). All hourly pressure data (both P₀ and P_z) associated with an R_z value that is greater than its tolerance are flagged for further analysis as a result of this hydrostatic check.

For each station, the tolerance of R_z is determined by the so-called sigma test (Hubbard et al. 2005; Shewhart 1980),

View Expanded

where μ and σ are the mean and standard deviation of the hourly R_z time series, respectively, and γ is a parameter that defines the tolerance in terms of σ. A value passes the hydrostatic check if the above relationship holds. Clearly, the tolerance range of R_z depends on the estimates of μ and σ. Generally, in the absence of data errors, both μ and σ should be near zero (cf. Fig. 2a). However, it was found that many data problems and errors could significantly bias the estimates of μ and σ. For example, as shown in Fig. 2b, a clear step was found in the time series of R_z on 3 October 1965, with most of the residuals (in absolute value) before 1965 equal to the station elevation (19.2 m), which is apparently an error caused by the 50-ft-rule problem (cf. section 3c). Because step changes in the R_z time series could significantly affect the estimates of μ and σ, and hence the R_z tolerance used in the hydrostatic check, we need to identify and correct step changes in the R_z series first, so that more realistic R_z tolerance can be determined for screening random errors. That is, we need to identify and correct systematic errors in the related pressure series first. We do so by combining a statistical homogeneity test with the hydrostatic check, which is the most innovative part of the QA system proposed here and is described later in section 4.

Once all the mean shifts (systematic errors) are identified and corrected, the mean and standard deviation (μ and σ) of the new R_z time series (calculated from the corrected pressure data) can be used in (4) to set the R_z tolerance for the screening of random errors. With the more accurate estimates of μ and σ, the value of γ in (4) can now be selected by predetermining the upper limit for the random error rate, as practiced in Hubbard et al. (2005). In this study, the upper limit of random error rate is set to 0.2‰ for all stations analyzed. That is, we cap the random error rate uniformly across the country, rather than using a fixed γ value; the values of γ are determined in such a way that an upper limit of 0.2‰ random error rate is kept for each and every station (thus, for a station with 50-yr hourly observations, there will be 87 data flagged for further investigation). Note that the uniform rate of 0.2‰ is used just as an upper limit of random error rate. The actual rate of random errors that are corrected as a result of this procedure does vary from station to station, and systematic errors are distinguished from random errors (they are identified and corrected first).

4. Identification of systematic errors

As shown in Fig. 2b and discussed earlier in section 3d, there are mean shifts in the hydrostatic residual R_z series, which reflect mean shifts (systematic errors) in the related pressure series.

Figure 3 shows two more examples of systematic errors in the hydrostatic residual and station pressure series. This type of error was found for many stations, especially the Arctic stations, and is not due to the 50-ft-rule problem. For some unknown reason (maybe an error in the archive data ingestion), the station pressure values for the period from 1992 up to 2002 were wrongly loaded for about 40 stations, including 18 Arctic stations. The associated R_z values (Figs. 3a,c) are incredibly high, showing a clear step change that would be easy to detect statistically, and most of the associated P_z values (see Figs. 3b,d) are unrealistically high and obviously wrong.

Although some systematic errors can be easily identified through visualization of the R_z and pressure series together (as shown in Fig. 3), it is common practice to detect them statistically. Considering the nature/physics of the R_z time series, for the vast majority of stations it is reasonable to assume that R_z has an independent identical Gaussian (IID Gaussian) distribution with mean μ and variance σ² under the null hypothesis of no systematic errors (note that this assumption is violated in the cases of elevated stations; details are given later in this section). Thus, testing whether or not there is a step change in the R_z time series for the period from N₁ to N₂ (1 ≤ N₁ < N₂ ≤ N; n = N₂ − N₁ + 1) is to test

View Expanded

against

View Expanded

where step size Δ = μ₂ − μ₁ ≠ 0 and ε_t denotes an IID Gaussian variable of mean zero and variance σ². In the first homogeneity test (i.e., at the beginning of the process), N₁ = 1 and N₂ = N. In the successive tests, either N₁ or N₂ or both of them are set to the changepoints identified in the previous test(s); that is, the time series is segmented at the newly identified changepoint and a successive test is applied to each new segment of the time series [N₁ and N₂ are the first and last data points of the segment being tested; see Wang and Feng (2007) for the details].

Here, detection of an undocumented step change can be done with the T_max statistic as in the standard normal homogeneity test (SNHT; Alexandersson 1986), or equivalently using the following F_max statistic:

View Expanded

where

View Expanded

and

View Expanded

Similar to those in Wang (2003) and Lund and Reeves (2002), the critical values of the F_max statistic here are obtained from 10 million simulations under H₀ for each series length n. Also, the F_c statistic above, which has an F distribution with (1, n − 2) degrees of freedom under H₀, can be used to assess significance of a documented step change (i.e., one that is supported by metadata) at time c. Both the F_max and F_c statistics, along with metadata (if available), are used in this study to identify R_z time series that have a significant step change. These R_z time series are further investigated, along with the related P_z and P₀ time series, to identify the cause and correct for the step change (via correcting the erroneous pressure values). Note that, in the absence of data error, the R_z time series should be random, with zero trend and no climate signal in general. These features of the R_z series render the use of reference series in the homogeneity test unnecessary. Actually, this is the case where the no-trend assumption of SNHT (and its equivalent tests, such as the one outlined in this paragraph) truly holds.

In general, both station relocation, without an update to the large change in station elevation for pressure reduction, and a change in observing instrument (e.g., sensor used in automatic stations) are often the causes for sudden changes in the mean of the R_z time series (and, hence, large μ and σ values). As shown in Fig. 4a, the R_z time series for the Lytton (British Columbia, Canada) station shows a clear step change on 1 July 1989, which was found to have arisen from a relocation of station with a decrease of 27.4 m in station elevation that was not accounted for in the calculation of station pressure from barometer readings (i.e., the elevation of the old site, which is 27.4 m higher than the elevation of the new site, was used in the calculation). This step change can also be identified from the original time series of P_z (Fig. 4b).

Figures 2c and 4a also show examples in which the assumption of IID Gaussian distribution for the R_z time series is violated. Specifically, the R_z time series exhibit a clear annual cycle (periodic variation). Our further investigation reveals that all such cases are associated with highly elevated stations (e.g., Old Glory Mountain, which has an elevation of 2347 m), which indicates that this very likely reflects a problem with the sea level pressure reduction (cf. Mohr 2004; Pauley 1998). The reduction of station pressure to mean sea level assumes a fictitious air column between the height of the station and the mean sea level. Usually, the air temperature decreases with increasing height from the surface; the rate of such a temperature decrease with increasing elevation is called the temperature lapse rate. However, the mean temperature of the fictitious air column is unknown, and is usually approximated in Canada by using a standard temperature lapse rate and T_dry (cf. Savdie 1982). Also, a plateau correction (i.e., a correction for the plateau effects) has been added since November 1976 for all stations in Canada (Savdie 1982) in an attempt to get approximately the same amplitude of the annual variation of sea level pressure at all stations, regardless of their elevation (WMO 1964; Mohr 2004). However, as a result of the standard pressure reduction method, including the plateau correction, misleading sea level pressure values can be obtained for high-altitude stations (Mohr 2004). We also notice that, in general, the plateau correction results in a slight increase in the variance of R_z for low-elevation stations (see period 1976–2001 in Figs. 2a,b), and a slight decrease for high-elevation stations (not shown). However, the discontinuity in variance is of a very small magnitude (±1 m of standard deviation of R_z, in general) when compared with the other systematic errors or with random errors we are trying to correct using this QA system. This study aims at correcting discontinuities in the mean, rather than in the variance (the later needs completely different statistical tests). Evaluation of the existing pressure reduction method and correction of the pressure reduction problem are beyond the scope of this paper. We need to be aware of this problem and keep in mind that a large R_z variance does not always correspond to discontinuities in the mean. We plotted and visually examined all of the R_z time series of large variance to determine the cause. To improve the validity of the IID Gaussian assumption for these time series for the homogeneity test, we estimate and remove the annual cycle from such a R_z time series before applying the homogeneity test to it (the series of departures from the annual cycle should be much closer to an IID Gaussian series).

As shown in Fig. 5, significant step change(s) in the R_z time series are found to have mainly arisen from either the 50-ft-rule problem (see those marked with a square), a long run of obviously wrong P_z values (see those marked with a circle), or a station relocation without updates to the changed elevation (see those marked with a triangle). The absolute values of the mean and standard deviation of R_z time series calculated from raw (uncorrected) pressure data, shown in Figs. 5 and 6a, respectively, are particularly large at the stations of obviously wrong P_z values, much larger than those at other problematic stations. In other words, the effects of these errors are much larger than those that are due to the 50-ft-rule problem or station relocation.

5. Correction of errors

Errors in meteorological data are very complicated and not easy to correct. Nevertheless, we should try our best not to reject, but to be able to correct erroneous data, especially for data-sparse regions (e.g., the Arctic region). An automatic error correction system is designed in the study.

It is highly desirable to know what caused the errors before we start to correct them. Table 2 lists the four types of errors that are most often found in our digital hourly pressure database, in addition to those that lead to a significant step change in the R_z time series. The vast majority of errors are of E1 and E2 (see Table 2). The E3 error is a profound problem in the Canadian hourly pressure data that were digitized from paper archives. In Canada, hourly pressure values used to be recorded (manually on paper) in tenths of hectopascal, and only the last three digits were recorded (e.g., “132” for a pressure of 10132, or “587” for 9587; unit: 0.1 hPa). The omitted base number (10 000 or 9000, or even 8000) needs to be added back during the digitization of our paper archives. Unfortunately, it is not always easy to determine which base number should be added, and the algorithm used to do so makes mistakes. This is why this type of error occurs and can be very hard (even impossible) to correct. This type of error sometimes persists for several hours or days, or even months (cf. Fig. 7), and can be mistaken as systematic biases caused by station relocation or instrument change, etc. Unfortunately, the same base number problem affects the National Centers for Environmental Prediction–National Center for Atmospheric Research (NCEP–NCAR) reanalysis dataset for the period from 1948 to 1967 (NCEP–NCAR 2006). Usually this type of error will not cause any exceedance of the climatological limits; these errors will only be detected by the hydrostatic check (combined with statistical homogeneity test) or by the PC check (i.e., the first and last erroneous data usually cannot pass the PC check). Therefore, it is sometimes impossible for us to determine which of P_z or P₀ is in error. A visual inspection of the time series segment often helps identify this type of error, which we do in this study.

6. Analysis of the corrected data series

The QA approach described above is applied to each station for both pressure levels. Corrected data are stored with their corresponding flags. However, a second iteration of the QA was run with corrected data in order to detect any wrong corrections or erroneous data that went undetected at the first run.

The rate of random errors identified for most stations (systematic errors that were corrected as described in section 5a were not counted here) is less than 1‰. Of more than 1.8 × 10⁸ hourly pressure data values (both levels) processed, approximately 4.1 × 10⁶ (or 2.3%) data values (including systematic errors) have been corrected. About 30% of the detected errors can be automatically corrected, while human–machine interactive correction is needed to correct the other 70%.

As shown in Fig. 6b, the standard deviation of the R_z time series calculated using corrected station and mean sea level pressure data is much smaller, showing a more organized pattern in comparison with Fig. 6a. Large values are now seen only at the elevated stations.

The hydrostatic check plays an important role in the whole QA system. About 50% of all of the errors detected using the QA system were identified through this check; more specifically, all of the detected systematic errors were identified by the hydrostatic check in combination with the statistical homogeneity test, plus 20%–30% of the detected random errors were identified using the hydrostatic relationship alone. Also, our results show that it is reasonable to assume that the hourly dry-bulb temperature data used in the hydrostatic check are correct. The hydrostatic method can also be helpful in detecting inhomogeneities in atmospheric pressure data caused by station relocation, observer change, and so on, as shown in Fig. 4.

7. Concluding remarks

To build a high-quality database for atmospheric pressure (at both station and sea levels) in Canada, we have developed a comprehensive QA system that includes the hydrostatic check combined with a statistical homogeneity test, which was applied to hourly pressure data recorded in the last 50 yr at 761 Canadian stations. The combination of a physically based model with a statistical test is shown to be very powerful in detecting both random and systematic errors in pressure data and provides physically based, more accurate estimates of the adjustment/correction needed.

The results show that there are serious systematic errors in the Canadian historical atmospheric pressure data and that random error(s) are present for almost every station. Systematic errors are found to be caused either by the use of wrong station elevation values in the reduction of barometer readings to station or sea level pressure values (e.g., the 50-ft rule or station relocation without updating the station elevation), by transposing/swapping station and sea level pressure values, or by mistakes made in the archive data ingestion or data recording/digitization processes (e.g., use of a wrong base number). Fortunately, a vast majority of these errors can be detected and corrected by the QA system with either an automatic or interactive correcting method. The corrected P₀ and P_z data should be much more reliable and better suited for various climate studies, including their use in producing a 100-yr reanalysis (Compo et al. 2006).

It is also noticed that the introduction of the plateau correction in 1977 and the digital barometer (Vaisala barometer) around 2001 appear to cause small discontinuities in the pressure variance, which are not corrected in this study. The current QA system is designed only to detect and correct random errors and discontinuities in the mean (mean shifts).