Article Type:
Research Article

New Categorical Metrics for Air Quality Model Evaluation

Daiwen Kang Atmospheric Sciences Modeling Division, Air Resources Laboratory, National Oceanic and Atmospheric Administration, Research Triangle Park, North Carolina

Search for other papers by Daiwen Kang in
Current site
Google Scholar
PubMed Close
,
Rohit Mathur Atmospheric Sciences Modeling Division, Air Resources Laboratory, National Oceanic and Atmospheric Administration, and National Exposure Research Laboratory, U.S. Environmental Protection Agency, Research Triangle Park, North Carolina

Search for other papers by Rohit Mathur in
Current site
Google Scholar
PubMed Close
,
Kenneth Schere Atmospheric Sciences Modeling Division, Air Resources Laboratory, National Oceanic and Atmospheric Administration, and National Exposure Research Laboratory, U.S. Environmental Protection Agency, Research Triangle Park, North Carolina

Search for other papers by Kenneth Schere in
Current site
Google Scholar
PubMed Close
,
Shaocai Yu Atmospheric Sciences Modeling Division, Air Resources Laboratory, National Oceanic and Atmospheric Administration, Research Triangle Park, North Carolina

Search for other papers by Shaocai Yu in
Current site
Google Scholar
PubMed Close
, and
Brian Eder Atmospheric Sciences Modeling Division, Air Resources Laboratory, National Oceanic and Atmospheric Administration, and National Exposure Research Laboratory, U.S. Environmental Protection Agency, Research Triangle Park, North Carolina

Search for other papers by Brian Eder in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Apr 2007
Collections:
NOAA/EPA Golden Jubilee
DOI:
https://doi.org/10.1175/JAM2479.1
Page(s):
549–555
Restricted access

Abstract

Traditional categorical metrics used in model evaluations are “clear cut” measures in that the model’s ability to predict an “exceedance” is defined by a fixed threshold concentration and the metrics are defined by observation–forecast sets that are paired both in space and time. These metrics are informative but limited in evaluating the performance of air quality forecast (AQF) systems because AQF generally examines exceedances on a regional scale rather than a single monitor. New categorical metrics—the weighted success index (WSI), area hit (aH), and area false-alarm ratio (aFAR)—are developed. In the calculation of WSI, credits are given to the observation–forecast pairs within the observed exceedance region (missed forecast) or the forecast exceedance region (false alarm), depending on the distance of the points from the central line (perfect observation–forecast match line or 1:1 line on scatterplot). The aH and aFAR are defined by matching observed and forecast exceedances within an area (i.e., model grid cells) surrounding the observation location. The concept of aH and aFAR resembles the manner in which forecasts are usually issued. In practice, a warning is issued for a region of interest, such as a metropolitan area, if an exceedance is forecast to occur anywhere within the region. The application of these new categorical metrics, which are supplemental to the traditional counterparts (critical success index, hit rate, and false-alarm ratio), to the Eta Model–Community Multiscale Air Quality (CMAQ) forecast system has demonstrated further insight into evaluating the forecasting capability of the system (e.g., the new metrics can provide information about how the AQF system captures the spatial variations of pollutant concentrations).

* Additional affiliation: Science and Technology Corporation, Hampton, Virginia

Corresponding author address: Daiwen Kang, Atmospheric Modeling Division, U.S. EPA, Mail Drop E243-03, Research Triangle Park, NC 27711. Email: kang.daiwen@epa.gov

This article included in the NOAA/EPA Golden Jubilee special collection.

Abstract

Traditional categorical metrics used in model evaluations are “clear cut” measures in that the model’s ability to predict an “exceedance” is defined by a fixed threshold concentration and the metrics are defined by observation–forecast sets that are paired both in space and time. These metrics are informative but limited in evaluating the performance of air quality forecast (AQF) systems because AQF generally examines exceedances on a regional scale rather than a single monitor. New categorical metrics—the weighted success index (WSI), area hit (aH), and area false-alarm ratio (aFAR)—are developed. In the calculation of WSI, credits are given to the observation–forecast pairs within the observed exceedance region (missed forecast) or the forecast exceedance region (false alarm), depending on the distance of the points from the central line (perfect observation–forecast match line or 1:1 line on scatterplot). The aH and aFAR are defined by matching observed and forecast exceedances within an area (i.e., model grid cells) surrounding the observation location. The concept of aH and aFAR resembles the manner in which forecasts are usually issued. In practice, a warning is issued for a region of interest, such as a metropolitan area, if an exceedance is forecast to occur anywhere within the region. The application of these new categorical metrics, which are supplemental to the traditional counterparts (critical success index, hit rate, and false-alarm ratio), to the Eta Model–Community Multiscale Air Quality (CMAQ) forecast system has demonstrated further insight into evaluating the forecasting capability of the system (e.g., the new metrics can provide information about how the AQF system captures the spatial variations of pollutant concentrations).

* Additional affiliation: Science and Technology Corporation, Hampton, Virginia

Corresponding author address: Daiwen Kang, Atmospheric Modeling Division, U.S. EPA, Mail Drop E243-03, Research Triangle Park, NC 27711. Email: kang.daiwen@epa.gov

This article included in the NOAA/EPA Golden Jubilee special collection.

Save
Cover Journal of Applied Meteorology and Climatology
Journal of Applied Meteorology and Climatology

  • Black, T., 1994: The new NMC Mesoscale Eta Model: Description and forecast examples. Wea. Forecasting, 9 , 265–278.

  • Byun, D. W., and J. K. S. Ching, Eds. 1999: Science algorithms of the EPA Models-3 Community Multiscale Air Quality (CMAQ) modeling system. U.S. Environmental Protection Agency Science Doc. EPA-600/R-99/030, 727 pp.

  • Byun, D. W., and K. L. Schere, 2006: Review of the governing equations, computational algorithms, and other components of the Models-3 Community Multiscale Air Quality (CMAQ) modeling system. Appl. Mech. Rev., 59 , 51–77.

    • Search Google Scholar
    • Export Citation

  • Hanna, S. R., 2006: A review of uncertainty and sensitivity analysis of atmospheric transport and dispersion models. Preprints, 28th NATO/CCMS Int. Technical Meeting on Air Pollution Modeling and Its Application, Leipzig, Germany, NATO/CCMS, 225–237.

  • Houyoux, M. R., J. M. Vukovich, C. J. Coats Jr., N. M. Wheeler, and P. S. Kasibhatla, 2000: Emission inventory development and processing for the Seasonal Model for Regional Air Quality (SMRAQ) project. J. Geophys. Res., 105 , 9079–9090.

    • Search Google Scholar
    • Export Citation

  • Kang, D., B. K. Eder, A. F. Stein, G. A. Grell, S. E. Peckham, and J. McHenry, 2005: The New England air quality forecasting pilot program: Development of an evaluation protocol and performance benchmark. J. Air Waste Manage. Assoc., 55 , 1782–1796.

    • Search Google Scholar
    • Export Citation

  • Otte, T. L., and Coauthors, 2005: Linking the Eta Model with the Community Multiscale Air Quality (CMAQ) modeling system to build a national air quality forecasting system. Wea. Forecasting, 20 , 367–384.

    • Search Google Scholar
    • Export Citation

  • Rao, S. T., I. G. Zurbenko, R. Neagu, P. S. Porter, J. Y. Ku, and R. F. Henry, 1997: Space and time scales in ambient ozone data. Bull. Amer. Meteor. Soc., 78 , 2153–2166.

    • Search Google Scholar
    • Export Citation

  • Rogers, E., T. L. Black, D. G. Deaven, G. J. DiMego, Q. Zhao, M. Baldwin, N. W. Junker, and Y. Lin, 1996: Changes to the operational “early” Eta analysis/forecast system at the National Centers for Environmental Prediction. Wea. Forecasting., 11 , 391–413.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 2941 2607 1231
PDF Downloads 206 69 5