In the most comprehensive pyrheliometer comparison known to date, 33 instruments were deployed to measure direct normal solar radiation over a 10-month period in Golden, Colorado. The goal was to determine their performance relative to four electrical-substitution cavity radiometers that were calibrated against the World Radiometric Reference (WRR) that is maintained at the World Radiation Center in Davos, Switzerland. Because of intermittent cabling problems with one of the cavity radiometers, the average of three windowed, electrical-substitution cavity radiometers served as the reference irradiance for 29 test instruments during the 10-month study. To keep the size of this work manageable, comparisons are limited to stable sunny conditions, passing clouds, calm and windy conditions, and hot and cold temperatures. Other variables could have been analyzed, or the conditions analyzed could have employed higher resolution. A more complete study should be possible now that the instruments are identified; note that this analysis was performed without any knowledge on the part of the analyst of the instruments’ manufacturers or models. Apart from the windowed cavities that provided the best measurements, two categories of performance emerged during the comparison. All instruments exceeded expectations in that they measured with lower uncertainties than the manufacturers’ own specifications. Operational 95% uncertainties for the three classes of instruments, which include the uncertainties of the open cavities used for calibration, were about 0.5%, 0.8%, and 1.4%. The open cavities that were used for calibration of all pyrheliometers have an estimated 95% uncertainty of 0.4%–0.45%, which includes the conservative estimate of 0.3% uncertainty for the WRR.
A fundamental measurement of the Baseline Surface Radiation Network (BSRN; Ohmura et al. 1998) was to be the direct solar radiation using a windowless absolute cavity radiometer. This instrument is superior to typical, commercial all-weather pyrheliometers with a precision of around 0.2% when compared under favorable conditions to the World Radiometric Reference (WRR; Rüedi and Finsterle 2005). The WRR is maintained at the World Radiation Center in Davos, Switzerland (Fröhlich 1991). Given that the absolute uncertainty of the WRR itself is conservatively estimated at 0.3% (Fröhlich et al. 1995), it is implied that transferring the WRR to open cavity radiometers can yield an overall 95% uncertainty of between 0.4% and 0.45% for these radiometric measurements.
Operating windowless cavities under all weather conditions proved difficult because of wind and precipitation interference; only one BSRN site (Payerne, Switzerland—operated by MeteoSwiss) has succeeded in long-term measurements with a windowless cavity. That site has operated continuously since 1994; even this instrument is shuttered during rain or snow events. All other BSRN sites use all-weather pyrheliometers based on thermopile detectors that have been calibrated relative to WRR-traceable reference cavities. According to the manufacturers, these instruments have a measurement uncertainty of 1% (assumed to be one standard deviation). Therefore, their 95% uncertainties would be about twice this, but in practice no one is certain of how well they actually perform in a field setting with all weather conditions possible because the calibrations and typical evaluations are performed under nearly ideal conditions (Finsterle 2006). Although direct solar measurements have the lowest fractional uncertainty, in absolute irradiance (W m−2) direct solar measurements contribute the largest uncertainty to the measurement of shortwave irradiance. It is, therefore, of fundamental importance to quantify the uncertainty of these all-weather pyrheliometers for the variable conditions during which they operate. Because most of these instruments are fundamentally thermal devices, variable thermal conditions of operation could produce different uncertainties in the observations than what would be obtained from calibration during stable, clear-sky conditions.
Traditionally, pyrheliometer comparisons such as the International Pyrheliometer Comparisons (IPCs) (held every five years in Davos, Switzerland to compare to WRR) or the National Renewable Energy Laboratory (NREL) Pyrheliometer Comparisons (NPCs) (held annually except in IPC years in Golden, Colorado, to compare to cavity radiometers with direct traceability to WRR) are conducted under the very best observing conditions with no clouds, calm winds, and irradiances exceeding 700 W m−2. This experiment was different in that measurements were made continuously from November 2008 until September 2009 for all conditions experienced at a high-elevation, continental, midlatitude site. The conditions ranged from clear to cloudy, dry to rain or snow, cold to hot, and calm to windy. The performance of commercially available pyrheliometers under all of these conditions will be discussed in this paper.
2. Experimental setup
This experiment, dubbed the Variable Conditions Pyrheliometer Comparison (VCPC), was organized under the auspices of the BSRN and conducted at the Solar Radiation Research Laboratory (SRRL) of the NREL in Golden, Colorado (latitude 39.742°N, longitude 105.178°W, 1829 m). Measurements of temperature, humidity, wind speed, and direction were collocated near the pyrheliometers. Figure 1 includes all of the 29 pyrheliometers and four windowed cavities mounted on four automatic solar trackers. One of the windowed cavity radiometers functioned much of the time, but was frequently affected by a bad cable that allowed moisture to adversely impact the signal; this was not discovered until late in the process, so rather than deal with this intermittent behavior, it was not used for the analysis. Participation in the comparison was solicited from all current manufacturers of pyrheliometers known to the experimental team. To the extent possible, triplicate copies of a given make and model were included. Where necessary or possible, additional instruments were supplied by field researchers. There were seven instruments represented in triplicate and the remaining eight were either prototypes or extras from some of the triplicate groups. The triple redundancy improved representativeness of a given model performance. Over the course of the field phase, instrument data streams were identified only by serial number. During the analysis phase, the identity of specific instruments’ data streams were deliberately obfuscated with simple A, B, C, etc., labels; the make, model, and serial number of test instruments were not revealed to the analyst until the reviewers accepted the paper for publication.
The instruments were inspected and cleaned 5 days week−1, and, occasionally, but not routinely, on weekends, as would be the typical protocol at a BSRN site. Sampling was every 2 s with averages and standard deviations computed over 1 min.
The trackers performed well over the course of the experiment with virtually no downtime. The weather conditions at the test site range from typically hot, dry, southwest U.S. summertime to cold, snowy, midcontinent wintertime. Further details on the experimental setup and routine operating practices during the experiment will be published in S. Wilcox et al. (2010 unpublished manuscript).
Four unwindowed cavity radiometers that routinely participate in the World Radiation Center’s quinquennial IPCs in Davos, Switzerland, were used to provide a calibration for all of the pyrheliometers including the windowed cavities. This calibration was performed on 10 separate occasions throughout the study period. It was determined that none of the test instruments’ calibration values changed significantly over the 10-month period; that is, calibration results were within the estimated measurement uncertainty. The calibration assigned to each pyrheliometer was based on all of the simultaneous measurements made during these calibration events with the sun within ±0.3° of a 45°solar zenith angle.
Cavity radiometers (Kendall and Berdahl 1970; Brusa and Fröhlich 1986) are configured to allow direct sunlight to pass through an aperture of known diameter into a blackened cavity comprising a cone-shaped detector at the base of a cylindrical receiver to prevent light from escaping; this produces a thermopile response since the cavity is in thermal contact with the hot junction of a thermopile while the cold junction is in contact with a mass within the radiometer that is near ambient temperature. To perform a self-calibration of these instruments, sunlight is then blocked and a measured electric current heats the previously irradiated cone to the same temperature to produce the same thermopile response as when sunlight heated the cone. The power to heat the cone and the area of the aperture define the irradiance in watts per meter squared. This electrical calibration of the cavity radiometers was performed every 20 min day and night for all comparisons. These frequent electrical calibrations eliminate any sensitivity that may be associated with the thermopile response dependence on temperature. The average of three windowed cavity pyrheliometers was considered the best estimate of the direct beam irradiance to which all other pyrheliometer measurements were compared during the 10 months of comparisons.
The analysis was performed without knowing the identity of the manufacturers of individual test pyrheliometers. Given the history of some interaction between some experimental team members and some of the instrument suppliers, this prevented the formation of a conscious or unconscious bias that might have influenced the results. The analyst knew that there were seven sets of three identical pyrheliometers and which three instruments (identified by a letter) constituted a group. It was further revealed to the analyst that two of the eight prototypes were novel instruments that were not expected to perform well as, we shall see, was the case. The windowed cavity radiometers were not specifically intended to be under test since they were the WRR-traceable reference that was used for the comparison standard for the remaining instruments. The analyst, therefore, knew which were the windowed cavities, although their identity was obvious from their superior performance. Table 1, which identifies all instruments by manufacturer, model, and serial number, was added only after the reviewers accepted the paper for publication.
All of the plots to be shown in this section are the average irradiance as measured by three windowed cavity radiometers subtracted from the 1-min average data from each of the 29 test pyrheliometers; that is, a positive result means that a test pyrheliometer reads high relative to the cavity average and vice versa.
Data were discarded before the analysis began when any windowed cavity differed from either of the other two cavities by more than 20 W m−2. In addition, data were discarded if any pyrheliometer reading differed by more than 40 W m−2 from the average of the windowed cavity radiometers. This screening process removed less than 1.7% of the data. Many of the points removed were data taken during the window cleaning that occurred five times a week. Accurate cleaning times were not recorded; therefore, this procedure was adopted to remove those points. Most of the other data removed were likely associated with snow, rain, bird droppings, or other unidentified obstructions.
Some of the pyrheliometers were supplied with their manufacturers’ recommended temperature corrections; if they were provided, they were applied. About half of the pyrheliometers that were loaned for this study did not have postacquisition temperature corrections, as indicated in Table 1. Most manufacturers design instruments that include temperature compensation via an electric circuit to modify the output of the thermopile as a function of temperature. The additional corrections provided by some manufacturers are for the residual temperature response that is not totally corrected by this temperature compensation circuitry. The paper by Vignola and Reda (1998) provides one example of the temperature dependence of one type of pyrheliometer.
Figure 2 illustrates three statistical distributions of data that demonstrate the need for an unconventional analysis. A random number generator sampling from a normal probability distribution produced the black points. Note the symmetry of the data and the monotonic falloff from the center of this normal probability distribution. The gray and white distribution plots are based on 1-min differences of one of the thermopile pyrheliometers from the mean of the windowed cavity radiometers (gray) and 1-min differences of one of the windowed cavities from the mean of the windowed cavities (white). In both instances the distributions are asymmetric, most notably the gray one, but the white one tails off nonmonotonically on the left-hand side of the distribution. This and other tests for normally distributed data demonstrate it is not possible to use standard statistical calculations of uncertainty assuming normal distributions to describe the measurement uncertainty.
Instead, to establish the degree of agreement between each pyrheliometer and the three reference cavity radiometers, we took the differences of their 1-min averages throughout the entire test period and calculated the quantiles of these differences at 2.5% and 97.5% along with the mean, and the median of the differences. Figure 3 illustrates the presentation of the data that will follow using this methodology. The top and bottom horizontal bars of the uncertainty symbol in the “E” partition are the 2.5% and 97.5% quantiles, therefore, 95% of the differences are contained within these limits; the black dot is the median value for the differences, and the gray dot is the mean value of the differences. There will be three symbols, one for each instrument, in each of the partitions labeled Cav, A … G for the three reference windowed cavities and seven sets of triplicate test instruments. There are eight single pyrheliometers (P) (nominally for prototype, although this designation does not strictly fit all P instruments; a few are extra pyrheliometers from the seven groups of matched test pyrheliometers). The differences, which are always given as test pyrheliometer minus three-cavity average, are expressed in watts per meter squared.
Table 2 lists the environmental conditions that will be examined in plots similar to Fig. 3. The columns are the environmental condition label, the criteria used to specify those conditions, the number of 1-min values satisfying those conditions, and the average direct irradiance for those conditions.
The upper plot in Fig. 4 illustrates the nighttime data, defined as points with the solar zenith angle greater than 91.2°. Occasionally, on very clear mornings, anomalous refraction and the elevated experiment location above the eastern horizon permitted direct beam radiation to be detected at zenith distances up to this angle. The lower plot of Fig. 4 contains the data at smaller solar zenith angles than this; that is, with the sun above the visible horizon, but restricted to direct beam measurements under 4 W m−2 and intended to represent cloudy conditions. Since nighttime data resulting from instrumental noise can approach 4 W m−2 in some pyrheliometers, this cutoff was chosen rather than 0 W m−2 for cloud-blocked direct beam. The plots are similar with most means and medians within 1 W m−2, and 95% of the data within 1–2 W m−2. However, the A, D, and E groups plus P5 and P8 show noticeably larger variability. These plots illustrate each instrument’s response to the environment in the absence of the intended measurement (direct beam radiation). Ideally, such response would be zero.
Figure 5 is a summary of all daytime data. The mountainous western horizon at the measurement location blocks direct sunlight beyond a solar zenith angle of approximately 85°. The mean value of the direct was 410 W m−2. The instruments identified as P1 and P2 are obvious outliers. All aspects of their variability, including their different means and medians and their large uncertainties, make them unsuitable for climate measurements, and further analysis or comment on their results will not given; these two instruments will be included in the plots, however.
There are three fairly obvious groupings that can be made from the remaining 30 instruments. The three windowed cavities have the smallest range of variability with 95% of all data within 2 W m−2. Of course, we are comparing each windowed cavity with the mean of the three, but there are no large differences among the three. However, any systematic errors common to all three cavity radiometers would be masked by this approach, but no such artifacts were apparent in the inspection of comparisons with the test instruments (e.g., identical vacillations in results from all other instruments that could then be attributed to the reference cavities). Groups B, C, F, and G plus the prototypes P4, P6, and P7 have larger variability than the cavity radiometers, but their means and medians are almost the same, and they have minor positive biases of about 1–2 W m−2 with respect to the mean cavity values; 95% of the data are within 7 W m−2 of these mean values—for future discussion we refer to these as B type. The 1–2 W m−2 biases (which are fractionally very small relative to full scale measurements) could be attributed to atypical calibration conditions rather than measurement error of the instruments. Groups A, D, and E plus P5 and P8 show more asymmetry with mean and medians showing less overlap and 95% of the data within 18 W m−2 of the mean or median—we will refer to these as A type. P3 falls between the two groups. It should be noted that B3 and D3 data do not cover the whole period since the original B3 unit malfunctioned and a substitution was made on 24 February 2009; D3 had subtle intermittent cabling issues that were not resolved until after 28 May 2009.
In Fig. 6 points were selected because of their high variability within the 1-min average. Clouds passing through the field of view of the instrument during the minute were the source of the variability. There is a positive bias in all of the means and medians such that the test pyrheliometer reads higher than the 3-cavity average. B-type pyrheliometers have a 2–4 W m−2 bias and 95% of the data within 8 W m−2 of the mean (or median). A-type pyrheliometers have a bias in the 4–10 W m−2 range with 95% of the data spanning at least twice the range of the B-type instruments from their means or medians. Again, P3 has an intermediate behavior. The source of the positive biases in all of these pyrheliometers is not understood. Two possible sources for this bias are the response time and the fields of view of the instruments. Since the instruments were not identified during this study, further analysis was not attempted for this paper. If, as is the case for most thermopile pyrheliometers, their responses are slower than the cavity radiometers used here, the signal would drop more quickly in the cavity for a passing cloud, but rise more quickly as the cloud departed. This should be a symmetric process on average; hence, no bias is expected. If test instruments have a larger field of view than the 5° cavities, then passing clouds would be detected for a longer period using an instrument with a 5.7° field of view, but the bias would be of the opposite sign seen here, and there would be no effect from this in a pyrheliometer with a 5° field of view.
Figure 7 contains two plots: one for clear skies with direct irradiance over 700 W m−2 and one for very clear skies that would satisfy the criteria for IPCs or NPCs, that is, irradiance over 700 W m−2 plus little variability in the direct beam. There are subtle differences between top and bottom figures with slightly less variability (generally, smaller ranges) in the bottom plot. The bottom of Fig. 7 represents the best conditions for comparing pyrheliometers in terms of lowest uncertainty. B-type instruments again held onto their higher rating in these plots; however, prototype P5, which has performed like A-type instruments in previous plots, performed as well as the B types in the bottom of Fig. 7.
Figure 8 contains plots that indicate the effect wind and related heat transfer disruptions have on pyrheliometers. The top plot (calm) is similar to Fig. 7 for all clear conditions, but the bottom plot (windy) is clearly perturbed from this. Often, but not always, the wind seemed to produce negative offsets compared to calm conditions, and the instruments within groups showed more independent behavior with respect to bias, while the 95% ranges remained similar within the groups. The test instruments were mounted to ensure that like instruments had different positions on the trackers; cavity radiometers were placed on a single tracker. The close packing of the instruments seen in Fig. 1 could possibly bias these results in that some shielding from the wind is possible. This could be addressed in a follow on study now that the instruments and their tracker positions are identified.
Figure 9 contains plots of extreme temperature conditions with the top plot for temperatures below 0°C and with irradiances greater than 700 W m−2. In the bottom plot the temperatures were in excess of 30°C for irradiances exceeding 700 W m−2. The results show somewhat subtle, but erratic, changes. Recall that these instruments are temperature corrected electrically and, if a temperature correction was provided for the pyrheliometer, mathematically also. There is no consistent pattern of instruments reading high or low relative to cold and hot conditions, generally, or even within a group. The D3 only had readings between May and September 2009, as noted earlier; therefore, there are no measurements below 0°C for this pyrheliometer.
Figure 10 further investigates the temperature effects. In this plot no postacquisition corrections for temperature were applied. Some have electrical temperature corrections included in the thermopile circuit, but it is understood that in some cases these circuits are not connected if the tested temperature behavior is within acceptable bounds. In this figure, for example, the G group indicates little change in response with temperature, while the F group changes dramatically with temperature. Applying the temperature corrections provided by the manufacturer in post processing allows the F group to perform among the best of the noncavity pyrheliometers (see Fig. 9). The effects on other groups that have temperature corrections applied in the data processing show less dramatic effects from the corrections, usually, but not always, improving the outcome.
Figure 11 contains plots for one windowed cavity, one A-type pyrheliometer, and one B-type pyrheliometer. There are two plots for each: the first is a plot of the differences over the entire record as a function of solar zenith angle (SZA) and the second is as a function of temperature with temperature corrections applied. The gray line is a numerical smoother through the data to guide the eye through the high density of points. The windowed cavity radiometer has no SZA dependence and no temperature dependence, and the spread of points is small. Consider again that any systematic errors or behavior common to all three cavities in this regard would be masked by this approach. The A-type pyrheliometer shows some dependence on SZA and almost no temperature dependence until 15°C where the response dips slightly above this temperature. The spread of points is the largest of the 3 types. The B-type pyrheliometer has no SZA dependence and no temperature dependence until 15°C where it dips slightly at higher temperatures, as did the A-type instrument. The spread is higher than for the cavity radiometer, but much smaller than the A-type pyrheliometer. These plots are not shown for all instruments; however, most A- and B-type instruments had some solar zenith angle and temperature dependencies. These plots should be useful since, for example, it appears that improvements in temperature response could be made for higher temperatures for both the A- and B-type pyrheliometers in Fig. 11.
In Fig. 12, estimates of the 95% confidence intervals for a representative member of a group or for a single prototype instrument are given for the clear-sky conditions in the top of Fig. 7. These were calculated using the procedure in Phillips et al. (1997) for expressing uncertainty with an uncorrected bias. However, not all pyrheliometers have a bias for the clear-sky case in Table 2 for which the uncertainties are given in Fig. 12. The uncertainties are asymmetric in most cases. They were calculated using the following:
where U+ and U− are the upper and lower 95% uncertainties, 0.45% is the 95% maximum uncertainty of the unwindowed cavity radiometers used to transfer the calibration to the windowed cavities, ΔI± are the upper and lower 95% uncertainties associated with measurements of clear sky compared to the windowed cavities, and δ is the bias.
The uncertainties given in Fig. 12 are encouraging. While manufacturers typically state uncertainties of ±1% (1 standard deviation suggesting 2% for 95% confidence), all except P1 and P2 performed better than this. The windowed cavities’ uncertainties are only slightly larger than the uncertainty associated with open cavity measurements if the windowed cavities are calibrated with WRR-based unwindowed cavities. Cavity radiometers are on the order of 10 times the cost of thermopile pyrheliometers. B-type pyrheliometers performed consistently with 95% uncertainties better than 0.9%. A-type pyrheliometers had 95% uncertainties better than 1.7%. All of these statements are predicated on the calibrations of these instruments being tied routinely to WRR transfer calibrations over the course of the comparison. Note that the uncertainties will be larger for certain conditions examined here such as those made in windy and or cold conditions. Comparing Fig. 12 for the clear-sky cases and Fig. 5 for the general daytime cases gives a sense of the additional uncertainty for average conditions.
From Fig. 11, and other plots not shown in this paper, there are some improvements in uncertainty to be gained from applying temperature response corrections in addition to those that are electronically performed within the thermopile circuit or mathematically in postacquisition processing. A report on this comparison by S. Wilcox et al. (2010, unpublished manuscript) contains additional information on the temperature and solar zenith angle dependencies of all of the instruments in the study.
The P2 was operated without any direction on its operation; it is possible that some of the errant behavior of this instrument may be correctable. Although an attempt was made to include all major manufacturers of pyrheliometers, not all participated. How they compare will remain uncertain until a similar study is undertaken in the future. It is recommended that this study be repeated at some level of complexity every five years for new and improved pyrheliometers.
The Eppley Laboratory Inc., Hukseflux, Kipp and Zonen, and Middleton Solar are manufacturers of pyrheliometers who generously loaned some of the instruments used for this experiment.