1. Introduction
Air quality has been of growing interest in both the public eye and the scientific community. Air pollution has been extensively studied since the 1960s where, in California, urban growth and unique terrain are conducive to air pollution problems. Identifying these problems spawned work on predictive methods (McCollister and Wilson 1975; Aron and Aron 1978) in an attempt to forecast high pollution episodes, protect public health and environment aesthetics, and provide information regarding air pollution policy.
Health studies such as Krupnick and Ostro (1990) and Berry et al. (1991) indicate that prolonged exposure to air pollution can cause ocular irritation, reduced lung function, and general degradation of the respiratory system. It is only natural that the health-conscious public grows concerned as they become aware of the risks associated with polluted air. This, in turn, puts pressure on air shed managers to help mitigate emission of and exposure to air pollutants and their precursors. Strategies and programs aimed at reducing emissions and exposure to pollutants can be costly endeavors; therefore, air shed managers need forecasts to help them make accurate and timely decisions. These forecasts are often chosen based on their performance as measured by statistical quantities such as correlation, bias, and error. This study critically evaluates the sources of these forecasts by going beyond the standard evaluation methods and incorporating the decision-making aspects associated with air quality forecasting.
a. Air quality forecast systems
This paper focuses on assessing the value of forecasts produced by an operational numerical model and forecasts produced by human forecasters. The value of persistence forecasts are assessed for relative comparison. The model, human forecasters, and persistence forecasting are collectively referred to as forecast systems.
1) The National Air Quality Forecast Capability
The National Oceanic and Atmospheric Administration (NOAA) and the Environmental Protection Agency (EPA) developed the National Air Quality Forecast Capability [NAQFC; also known as the National Air Quality Forecast System (NAQFS) in previous literature] in partial fulfillment of the Energy Policy Act of 2002. The NAQFC couples the Weather Research and Forecasting Nonhydrostatic Mesoscale Model (WRF-NMM) (Janjic 2003) with the Community Multiscale Air Quality (CMAQ) model (Byun and Schere 2006) to produce 48-h forecasts of surface 1-h-average and 8-h-average ozone mixing ratios across the contiguous United States, Alaska, and Hawaii. The NAQFC also produces forecasts for smoke and will soon be adapted to produce forecasts for particulates; however, these will not be discussed here. Current NAQFC data are available through the National Weather Service (NWS) National Digital Guidance Database (http://www.weather.gov/aq), and archived NAQFC data are available through NOAA’s National Operational Model Archive and Distribution System (NOMADS).
The NAQFC has been subjected to rigorous verification (Ryan et al. 2004). Eder et al. (2006) performed the standard model evaluation statistics, such as bias and error analyses, in part to improve the NAQFC performance and prepare the NAQFC for operational use in 2005. Eder et al. (2009) evaluated the model using both standard and categorical statistics, such as false alarm rates, hit rates, and critical success indices. Eder et al. (2010) use areal forecast statistics in the latest NAQFC verification rather than point-to-point comparisons as done in the previous evaluations. These categorical statistics assess the model performance in a more typical air quality forecasting scenario where a forecast is issued for an area rather than a specific site. Their results indicate that the NAQFC forecast skill is comparable to that of expert human air quality forecasters and that the NAQFC will become more useful under stricter air quality standards.
2) Human forecasters and persistence forecasting
Air quality forecasters typically develop their own tools and algorithms to forecast pollution episodes in their respective areas of interest (Aron and Aron 1978; Lin 1982; Robeson and Steyn 1990; Ryan 1995; Hubbard and Cobourn 1998; Davis and Speckman 1999). Human air quality forecasters, however, offer real-world experience in blending standard information and novel interpretations of nontraditional data, such as human behavior and current events, into forecast algorithms.
A human air quality forecast for the following day in the mid-Atlantic region is submitted to AIRNow Tech by 1500 local time. The time to prepare a forecast can be limited because of the forecaster’s responsibilities outside of producing an air quality forecast. The ozone forecast is for the maximum 8-h average surface ozone mixing ratio for the day. A forecaster uses a wide array of data and information to create a forecast, and each forecaster has many preferred sources of information. Although there is no regular procedure to use, many forecasters often use similar sources. A human air quality forecast would typically include an analysis of the most recent meteorological-model simulations, air quality observations from sites within the forecast region and neighboring areas, and oftentimes a personally developed statistical model or empirical forecast rule (W. F. Ryan 2009, personal communication). The human forecasters do have access to the NAQFC when producing a forecast; however, it is one source of information, among many, that the human forecaster considers.
Persistence forecasting is a simple and straightforward forecast method. The idea is to use the current conditions as a forecast for the next day. Persistence forecasting is often regarded as a primitive forecast method, but it can be useful under specific (i.e., persistent, hence the name) circumstances. Poor air quality events are typically episodic, which is why persistence can be considered a good benchmark forecast, which is then adjusted according to information at hand.
b. Air quality and decision making
1) Ozone and the air quality index
Ozone is an abundant oxidant in the atmosphere. Near-surface ozone, however, has been extensively studied and found to be a harmful pollutant (Lippmann 1989; Wright et al. 1990; Berry et al. 1991; Chen et al. 2007). This ozone is produced by the photolysis of nitrogen oxides (NOx) and oxidation of volatile organic compounds (VOCs) (Seinfeld and Pandis 2006). NOx is primarily produced by combustion processes such as natural fires, vehicles, and power plants. VOCs are produced by plant life, by burning of carbon-based fuels, and in thousands of industrial processes such as plastics manufacturing and smelters. The EPA designated ozone as a criteria pollutant because of its heavy dependence on anthropogenically produced precursors and malicious effects on human health. As a criteria pollutant, ozone is subject to restrictions outlined in the National Ambient Air Quality Standards (NAAQS; http://www.epa.gov/air/criteria.html).
Currently, six criteria pollutants are monitored by state agencies and reported to the EPA as part of the Clean Air Act. The concentration of these criteria pollutants is made available to the general public via an air quality index (AQI). As discussed in Mintz (2009), the AQI is a method of normalizing pollutant concentrations into a single number to convey the level of threat to public health. The AQI is calculated using a standard linear interpolation from the pollutant concentration to the index using the breakpoints in Table 1. For convenience, the AQI is color coded to enable quick interpretation of the health risk. Poor air quality is generally associated with an orange AQI (AQI = 101) or higher. Of the criteria pollutants, particulates and ozone routinely reach unhealthy levels in many urban areas throughout the United States. This study will focus on ozone in the mid-Atlantic region, where elevated ozone is often observed, and a variety of protective measures are employed by local and regional governments.
The AQI with associated health risk and color code. Breakpoints in parentheses are for dates prior to March 2008, when the ozone standards changed. Adapted from Mintz (2009).
2) Costs associated with air quality
Air shed managers need to consider a variety of direct and indirect costs associated with air quality. When ozone is forecasted to be above the NAAQS threshold (AQI > 100) on a given day, an “alert day” may be invoked with special programs. The implementation of such programs is designed to reduce the emissions of the ozone precursors and avoid losses due to high levels of ozone such as health costs and EPA nonattainment (Anderson 2001). Accurate ozone forecasts are needed to make proper decisions regarding program implementation. Inaccurate forecasts could lead to unneeded action or missed opportunities to mitigate ozone precursor emissions, which both would result in preventable monetary loss.
2. Data
Ozone observations were assembled from the Air Quality System (AQS), a database hosted by the EPA Technology Transfer Network with contributions from local, state, and federal agencies. The data specifically used in this study were provided to the AQS by the Virginia Department of Environmental Quality (VADEQ), the Maryland Department of the Environment (MDE), and the District Department of the Environment (DDOE). Average hourly surface ozone mixing ratios were collected for the 2005–09 ozone seasons (April–October) at 40 locations throughout Maryland; Washington, D.C.; and Virginia. The 1-h averages were converted to hourly 8-h forward running averages, which were then used to calculate the AQI. The breakpoints for calculating ozone AQI were reduced in 2008; therefore, AQI was calculated using the old breakpoints for the 2005–07 ozone seasons and the new breakpoints in the 2008 and 2009 ozone seasons. The maximum AQI for a given day among all the sites within a region would represent the observed AQI for that region on that day, similar to the methods used in the second evaluation approach in Eder et al. (2010). Refer to Fig. 1 for a map color coded by region.
A map of the sites within each forecast region. The dots represent individual sites, and the colors represent forecast regions. Note that some sites do overlap.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-10-05010.1
NAQFC forecast 8-h average surface ozone mixing ratios were obtained from NOMADS for the same time period and sites as the observations and were converted to AQI. The maximum AQI out of all the sites within a region is used for the analysis. Human air quality forecasts of daily maximum AQI were also obtained directly from the VADEQ and MDE for the same time period. These forecasts were provided for the regions, not individual sites, so no further conversions or calculations were needed. Persistence forecasts were developed simply by using the maximum AQI for the current day as the forecast for the next day. Days missing either an observation or a forecast for any one of the forecast systems were excluded from this study.
3. Methods
Standard verification statistics were computed to assess the skill of each forecast system. We used those verification statistics set forth by Eder et al. (2010), including the mean bias (MB), normalized mean bias (NMB), root-mean-square error (RMSE), normalized mean error (NME), and correlation coefficient (R) of each forecast system as a whole and categorical statistics such as critical hit rate (cH), exceedance hit rate (eH), exceedance false alarm rate (eFAR), and exceedance critical success index (eCSI) for each forecast system within each region. These categorical statistics are explained in greater detail in Eder et al. (2010).
A static cost–loss ratio model was used to assess the value of each forecast system. This model was first discussed in a meteorological context in Thompson (1952) and Thompson and Brier (1955). Kernan (1975) is an early example of applying the model to an air pollution decision-making situation. The cost–loss ratio model is discussed in detail in the Thompson papers; the Kernan paper; and more recently in Katz and Murphy (1997), Richardson (2000), Thornes (2001), and Berger (2006). Recent examples of the application of the cost–loss ratio model can be found in Rotach et al. (2009) and Millner (2009).
The method used to calculate value in this study is detailed in Richardson (2000) and is summarized here. Start with a 2 × 2 contingency table that compares the cost C to protect or insure against a loss L if a poor air quality event were to occur (Table 2). Often, C and L are expressed as some form of currency. Examples of C include free bus rides, reduced production at power plants, and other measures aimed at reducing pollutant and pollutant precursor concentrations. Examples of L include loss of tourism, environmental degradation, and an increased number of patients in hospital emergency rooms. Insuring against a loss will cost C, whether or not the poor air quality event occurs; however, one would lose L if the poor air quality event occurs and no protective measures were implemented. One assumption in this model is that C < L, which makes sense, because there would be no decision to make otherwise.
Contingency table indicating the cost C to protect against a loss L depending on the state of the air quality. There is neither cost nor loss when no protective measure is taken on a good air quality day.
Now consider N number of cases, with each case having an air quality forecast and observation. Here, Nf is the number of times the event was forecasted but not observed, No is the number of times the event was observed but not forecasted, and Nfo is the number of correctly forecasted poor air quality events. The climatological frequency of these events is defined as s = (Nfo + No)/N. Expected losses can be formulated using Table 2 and the definitions above.
4. Forecast system verification
a. Overall
The scatterplots in Fig. 2 depict the discrete statistics and overall skill of the human forecaster (top left), the NAQFC (top right), and persistence (bottom). The human forecaster outperforms both the NAQFC and persistence according to the correlation coefficient (0.73 compared to 0.65 and 0.58, respectively), RMSE (18.3 compared to 25.2 and 23.4, respectively), and NME (23.9% compared to 33.3% and 30.1%, respectively). The NAQFC is heavily positively biased (MB of 10.97 and NMB of 20.57%), whereas both the human forecasters and persistence forecasting are only slightly positively biased. Although the human forecasters have a slight positive bias, a preponderance of the observation–forecast pairs fall below the 1:1 line, indicating conservative forecasting.
Scatterplots with discrete statistics for (top left) the human forecaster, (top right) the NAQFC, and (bottom) persistence forecasting over all the forecast regions. Each dot represents an observation–forecast pair.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-10-05010.1
Confidence intervals were calculated on the R, RMSE, and MB discrete statistics using a bootstrap algorithm. The observation–forecast pairs were repeatedly subsampled with replacement, creating 10 000 bootstrap samples from which the 95% confidence intervals were derived. The distributions and confidence intervals of these bootstrap samples are shown in Fig. 3. The horizontal lines under the distributions indicate the 95% confidence interval. None of the confidence intervals overlap, implying that the differences between the forecast systems’ R, RMSE, and MB are statistically significant at 95%.
Empirical distributions with 95% confidence intervals of (top) R, (middle) RMSE, and (bottom) MB of each forecast system. Each distribution was developed using 10 000 bootstrap subsamples of observation–forecast pairs.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-10-05010.1
b. Regional and threshold specific
Categorical statistics were calculated for each forecast system within each region at the orange AQI threshold. Figure 4 describes the categorical statistics for the Virginia regions. The four panels show the cH (top left), eFAR (top right), eH (bottom left), and eCSI (bottom right) for each region within the state. Hampton Roads and Winchester recorded fewer exceedance days than the other six regions (see Table 3), explaining the low cH, eH, and eCSI and the high eFAR in Fig. 4. The human forecaster performs better than both the NAQFC and persistence in all regions and statistics but one. The NAQFC performed better according to the eH in northern Virginia; however, the strong positive bias in the NAQFC gives the model an advantage in this statistic. The human forecaster eCSI is only slightly higher than both the NAQFC and persistence eCSI in northern Virginia and approximately 10%–20% higher in Richmond. The high eFAR for both the NAQFC and persistence dramatically reduced their respective eCSI in these regions. The eCSI for the NAQFC and persistence in both Hampton Roads and Winchester is zero because the eH for both these forecast systems is zero in these regions.
Categorical statistics for all of the Virginia regions. Statistics include the (top left) cH, (top right) eFAR, (bottom left) eH, and (bottom right) eCSI.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-10-05010.1
Frequency chart used in categorical statistics calculations using the orange AQI threshold (AQI > 100). Here, N is the total number of observation–forecast pairs, No is the number of observations not forecasted above the threshold, Nf is the number of forecasts not observed over the threshold, and Nfo is the number of observations forecasted above the threshold.
The categorical statistics for the Maryland regions are depicted in Fig. 5 with a similar setup to Fig. 4. The NAQFC and persistence forecasting performs the same as (within 2%) if not better than the human forecaster in both the hit-rate statistics in all of the Maryland regions but Baltimore. Just as in the Virginia regions, the high eFAR for both the NAQFC and persistence forecasting drastically reduces their respective eCSI. The eCSI indicates that the human forecaster outperforms the NAQFC and persistence forecasting in Baltimore; Washington, D.C.; and Millington by 5%–15%; however, persistence forecasting performs the best in Hagerstown. This could be explained by the infrequent observations of ozone above the orange AQI threshold (Table 3) in Hagerstown and that one of the verification sites within the region is at an elevation of 764 m, making accurate forecasts difficult.
Categorical statistics for all of the Maryland regions. Statistics include the (top left) cH, (top right) eFAR, (bottom left) eH, and (bottom right) eCSI.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-10-05010.1
c. Value of forecast systems
The value of each forecast system was calculated with Eq. (5) at the orange AQI threshold, and these values are discussed in this section. The value of each forecast system was also calculated for the red and yellow AQI thresholds, but these values are not included in this study. There were too few days with observed ozone in the red AQI range to make a robust statistical analysis. Conclusions drawn from the yellow threshold analysis were consistent with the orange threshold analysis. Including these results would be redundant and have little meaning because alert days are rarely, if ever, invoked on a code yellow day.
Figure 6 shows the value curves for all the forecast regions in both Virginia and Maryland using the human forecaster (top left), the NAQFC (top right), and persistence (bottom left). The calculated value lies on the ordinate, and the ratio of the costs to losses lies along the abscissa of each of the value plots in Fig. 6. The value can be interpreted as the percent saved from the difference between no forecast system and a perfect forecast system. Assuming that L does not change, the cost–loss ratio may be interpreted as a given protective measure to insure against the loss. For example, the Baltimore value curve for the human forecaster peaks at 0.529 with a cost–loss ratio of 0.137 (Table 4). This means that the air shed manager would save 52.9% of the difference between the expense of no forecast system and a perfect forecast system when deciding to issue a protective measure costing 13.7% of the losses. If the losses total $100,000 in this situation, one would expect to save $6,254 per forecasted event [13.7% × ($100,000 − $13,700) × 52.9% = $6,254].
Value curves for (top left) the human forecaster, (top right) the NAQFC, and (bottom left) persistence forecasting. Value curves were calculated using an orange AQI threshold (AQI > 100).
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-10-05010.1
Table showing the peak value, the cost–loss ratio of the peak value, and the range of cost–loss ratios over which each forecast system holds value.
The value curves of the human forecasts cover a broader range of cost–loss ratios than either the NAQFC or persistence forecast value curves. Although the minimum cost–loss ratio at which each forecast system has positive value is similar among all the regions (between 0.01 and 0.1), the maximum cost–loss ratio over which the human forecast has positive value is 10%–30% higher than the NAQFC and persistence forecasts for their respective regions. This indicates that the human forecaster produces forecasts that can save air shed managers money over a wider range of protective measures than the other forecast systems. Among the forecast systems, the NAQFC forecasts produce the highest peak value in Washington, D.C. (V = 0.558) and Millington (V = 0.475), whereas persistence forecasts produce the highest peak value in Hagerstown (V = 0.237). The human forecasts produce the highest peak value in the remaining regions. Neither the NAQFC nor persistence forecasts produce value by our definition at Hampton Roads and Winchester, because neither forecast system correctly forecasted a poor air quality event during the analysis period (eH = 0).
With access to the three forecast systems discussed in this study, the air shed manager would be able to choose the forecast system that produces the highest value depending on the protective measure. The plots in Fig. 7 show the maximum value of the three forecast systems at each region as a function of the cost–loss ratio. These curves are a combination of the human, NAQFC, and persistence value curves and are color coded according to the forecast system that produces the highest value for that specific cost–loss ratio. Of the eight forecast regions, five benefit from a combination of forecast systems. In the Washington, D.C. region, the NAQFC produces the highest value up to a cost–loss ratio of 0.2, after which the human forecaster produces the highest value. In the Hagerstown region, persistence forecasts produce the highest value up to a cost–loss ratio of 0.3, after which the human forecast produces the highest value. The NAQFC technically produces the highest value at low cost–loss ratios in the northern Virginia, Baltimore, and Millington regions; however, the combined curves are not much different than the human forecast value curves.
Value curves for each forecast system separated by forecast region. The lines are color coded according to the forecast system. The dashed lines indicate the value of each forecast system. The bold lines indicate the forecast system, with the highest value at the given cost–loss ratio.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-10-05010.1
These results indicate that forecast skill does not directly translate to forecast value. The choice of forecast system is highly dependent upon the user’s particular need, and using an overall measure of skill, such as eCSI, RMSE, R, etc., can often hide this. In some decisions, the less-skillful forecast system may provide the highest value.
5. Summary and conclusions
Standard discrete and categorical statistics were used to evaluate human air quality forecasts, the NAQFC forecasts, and persistence forecasts in eight regions throughout Virginia and Maryland over five ozone seasons (2005–09). These statistics were supplemented with an assessment of value that incorporates a decision-making aspect into the forecast. The human forecasts performed better than both the NAQFC and persistence forecasts in the discrete statistics. The human forecaster, though slightly positively biased, typically produced conservative forecasts, whereas the NAQFC would consistently overforecast. The categorical statistics indicated that the human forecaster is the most skillful in all the regions but Hagerstown, where persistence forecasting is the most skillful. This is likely due to the elevation of the Hagerstown region and the infrequent occurrences of poor air quality events.
The value of each forecast system varies greatly upon the protective measure being considered. The NAQFC is able to produce higher-value forecasts than human forecasters at relatively low cost–loss ratios in urban and downwind regions; however, the human forecaster is able to produce high-value forecasts for a broader range of cost–loss ratios in all regions in this study. This implies that the most skillful forecast system may not provide the best value in all situations, and thus it is wise for air shed managers to consider multiple forecast systems when deciding on a number of protective measures.
These value metrics can be easily applied to other areas prone to poor ozone events, such as Houston or Los Angeles. These metrics are also being considered to evaluate optimum placement of monitoring sites within forecast regions. A value assessment on individual sites within a region may provide local air shed managers with useful information on their monitoring strategies. A series of case studies performed in a single forecast region, such as Baltimore or Washington, D.C., that attempt to monetarily quantify losses incurred during a poor air quality event will help tailor these value calculations to more specific decisions and assess the forecast systems’ value more accurately.
Acknowledgments
We would like to acknowledge the Metropolitan Washington Council of Governments, the District Department of the Environment, the Maryland Department of the Environment, and the Virginia Department of Environmental Quality for contributing their data to the Air Quality System and AIRNow Tech; William Ryan (Penn State) for vital information about the air quality forecasting process; and Duc Nguyen (MDE), Laura Landry (MDE), and Dan Salkovitz (VADEQ) for their help accessing the forecast data and comments regarding the study. We would also like to acknowledge the anonymous referees who volunteered their time to review this article and provide constructive and insightful feedback. This study was funded by NSF DRU Program Award 0729413 with added support from NASA’s Tropospheric Chemistry Program (J. H. Crawford and J. A. Al-Saadi) and Aura Validation (M. J. Kurylo and K. W. Jucks).
REFERENCES
Anderson, R. C., 2001: Pollution charges, fees, and taxes. The U.S. Experience with Economic Incentives for Protecting the Environment, Environmental Protection Agency National Center for Environmental Economics Rep. EPA-240-R-01-001, 33–55.
Aron, R., and Aron I. , 1978: Statistical forecasting models. I. Carbon monoxide concentrations in the Los Angeles Basin. J. Air Pollut. Control Assoc., 28, 681–684.
Berger, J. O., 2006: Basic concepts. Statistical Decision Theory and Bayesian Analysis, 2nd ed. Springer, 1–34.
Berry, M., Lioy P. J. , Gelperin K. , Buckler G. , and Klotz J. , 1991: Accumulated exposure to ozone and measurement of health effects in children and counselors at two summer camps. Environ. Res., 54, 135–150.
Byun, D., and Schere K. L. , 2006: Review of the governing equations, computational algorithms, and other components of the models-3 Community Multiscale Air Quality (CMAQ) modeling system. Appl. Mech. Rev., 59, 51–77.
Chen, T., Gokhale J. , Shofer S. , and Kuschner W. G. , 2007: Outdoor air pollution: Ozone health effects. Amer. J. Med. Sci., 333, 244–248.
Davis, J. M., and Speckman P. , 1999: A model for predicting maximum and 8 h average ozone in Houston. Atmos. Environ., 33, 2487–2500.
Eder, B., Kang D. , Mathur R. , Yu S. , and Schere K. , 2006: An operational evaluation of the Eta-CMAQ air quality forecast model. Atmos. Environ., 40, 4894–4905.
Eder, B., Kang D. , Mathur R. , Pleim J. , Yu S. , Otte T. , and Pouliot G. , 2009: A performance evaluation of the National Air Quality Forecast Capability for the summer of 2007. Atmos. Environ., 43, 2312–2320.
Eder, B., and Coauthors, 2010: Using national air quality forecast guidance to develop local air quality index forecasts. Bull. Amer. Meteor. Soc., 91, 313–326.
Hubbard, M. C., and Cobourn W. G. , 1998: Development of a regression model to forecast ground-level ozone concentrations in Jefferson County, Kentucky. Atmos. Environ., 32, 2637–2647.
Janjic, Z. I., 2003: A nonhydrostatic model based on a new approach. Meteor. Atmos. Phys., 82, 271–285.
Katz, R. W., and Murphy A. H. , 1997: Forecast value: Prototype decision-making models. Economic Value of Weather and Climate Forecasts, R. W. Katz and A. H. Murphy, Eds., Cambridge University Press, 183–217.
Kernan, G. L., 1975: The cost-loss decision model and air pollution forecasting. J. Appl. Meteor., 14, 8–16.
Krupnick, A. J., and Ostro W. H. B. , 1990: Ambient ozone and acute health effects: Evidence from daily data. J. Environ. Econ. Manage., 18, 1–18.
Lin, Y., 1982: Oxidant prediction by discriminant analysis in the south coast air basin of California. Atmos. Environ., 16, 135–143.
Lippmann, M., 1989: Health effects of ozone. A critical review. J. Air Pollut. Control Assoc., 39, 672–695.
McCollister, G. M., and Wilson K. R. , 1975: Linear stochastic models for forecasting daily maxima and hourly concentrations of air pollutants. Atmos. Environ., 9, 417–423.
Millner, A., 2009: What is the true value of forecasts? Wea. Climate Soc., 1, 22–37.
Mintz, D., 2009: Technical assistance document for the reporting of daily air quality: The Air Quality Index (AQI). Environmental Protection Agency Tech. Rep. EPA-454/B-09-001, 31 pp.
Richardson, D. S., 2000: Skill and relative economic value of the ECMWF ensemble prediction system. Quart. J. Roy. Meteor. Soc., 126, 649–667.
Robeson, S. M., and Steyn D. G. , 1990: Evaluation and comparison of statistical forecast models for daily maximum ozone concentrations. Atmos. Environ., 24B, 303–312.
Rotach, M. W., and Coauthors, 2009: Map D-PHASE: Real-time demonstration of weather forecast quality in the alpine region. Bull. Amer. Meteor. Soc., 90, 1321–1336.
Ryan, W. F., 1995: Forecasting severe ozone episodes in the Baltimore metropolitan area. Atmos. Environ., 29, 2387–2398.
Ryan, W. F., Davidson P. , Stokols P. , and Carey K. , 2004: Evaluation of the National Air Quality Forecast System (NAQFS): Summary of the air quality forecasters focus group workshop. Preprints, Sixth Conf. on Atmospheric Chemistry: Air Quality in Megacities and Symp. on Planning, Nowcasting, and Forecasting in the Urban Zone, Seattle, WA, Amer. Meteor. Soc., J2.13. [Available online at http://ams.confex.com/ams/pdfpapers/70567.pdf.]
Seinfeld, J. H., and Pandis S. N. , 2006: Chemistry of the troposphere. Atmospheric Chemistry and Physics: From Air Pollution to Climate Change, 2nd ed. John Wiley and Sons, 205–283.
Thompson, J. C., 1952: On the operational deficiencies in categorical weather forecasts. Bull. Amer. Meteor. Soc., 33, 223–226.
Thompson, J. C., and Brier G. W. , 1955: The economic utility of weather forecasts. Mon. Wea. Rev., 83, 249–254.
Thornes, J. E., 2001: How to judge the quality and value of weather forecast products. Meteor. Appl., 8, 307–314.
Wright, E. S., Dziedzic D. , and Wheeler C. S. , 1990: Cellular, biochemical and functional effects of ozone: New research and perspectives on ozone health effects. Toxicol. Lett., 51, 125–145.
