VOLUME30 JOURNAL OF APPLIED METEOROLOGY JULY 1991Evaluation of an Air Pollution Analysis System for Complex Terrain D. G. RossCentre for Applied Mathematical Modelling, Monash University, Melbourne, Australia D. G. FoxRocky Mountain Forest and Range Experiment Station, USDA, Fort Collins, Colorado(Manuscript received 17 July 1990, in final form 4 December 1990) ABSTRACT This paper describes results from a study to evaluate components of an operational air quality modelingsystem for complex terrain. In particular, the Cinder Cone Butte (CCB) "modeler's dataset" is used to evaluatethe current technique for incorporating terrain influences and atmospheric stability into the system's 3D diagnosticwind-field model. The wind-field model is used in conjunction with a Gaussian puffmodel to compare predicted and observedtracer concentrations for different configurations, chosen to highlight the influence of the model's technique forincorporating terrain and atmospheric stability in the final flow field. A quantitative statistical basis, includingthe use of a bootstrap resampling procedure to estimate confidence limits for the performance measures, is usedfor the evaluation. The results show that the model's technique for incorporating terrain and atmosphericstability yields a significant improvement in predictive performance. Even when only routinely available inputdata are used, the performance is shown to be as good as that of models based directly on the CCB data.set itself.1. Introduction Models to estimate air quality in complex terrainface a number of challenges in their development andapplication. There are many unanswered scientificquestions about both the mean and turbulent characteristics of airflow in complex terrain. As a result, anumber of major research efforts have been undertaken, which include the Complex Terrain Model Development (CTMD) program, with results reported asa series of"milestone" reports (e.g., DiCristofaro et al.1985); and the Atmospheric Studies in Complex Terrain (ASCOT) program, with results published as a setof papers in the June and July 1989 volumes of theJournal of Applied Meteorology. The CTMD programfocused on plume impaction--the dynamics of plumebehavior in the vicinity of an isolated topographic feature, while the ASCOT program focused on an understanding of plume behavior in nocturnal drainageflows. Both programs provide not only detailed datarequired to understand the physics of each flow type,but also databases for the development and evaluationof models. The topographic air pollution analysis (TAPAS) is aflexible computer modeling system being developed Corresponding author address: Dr. D. Graeme Ross, Centre forApplied Mathematical Modelling, Monash University, P.O. Box 197,Caulfield East, Melbourne, Victoria 3145, Australia.jointly by the Centre for Applied Mathematical Modelling (CAMM) at Monash University and the RockyMountain Forest and Range Experiment Station, U.S.Department of Agriculture--Forest Service. TAPAScontains simulation models of varying complexity andranges of applicability, input data management routines, and graphical display procedures designed to assist air resource managers. The details of the overallsystem are described elsewhere (e.g., Fox et al. 1987).Predicting the concentration of atmospheric pollutantsin complex terrain typically requires an air qualitymodeling system that is capable of simulating bothspatial and temporal variations in meteorology andturbulent diffusion. The TAPAS system configured touse the NUATMOS and CITPUFF models representsa cost-effective system capable of simulating thesevariations in complex terrain, while maintaining muchof the simplicity of the basic Gaussian approach. CITPUFF (Ross et al. 1987) is a multiple-sourcepuff model capable of taking as its transport field afully three-dimensional (3D) wind field. The currentversion is designed to accept wind fields from the complex terrain diagnostic wind field model NUATMOS(Ross et al. 1988a,b; Smith and Ross 1988); however,modification to accept the output of an alternative windfield model is straightforward. The models have undergone an extensive testing and evaluation program,both as separate models and in combination (e.g., Lorimer 1989). The program has included comparison ofmodel predictions with exact solutions, data from lab909910 JOURNAL OF APPLIED METEOROLOGY VOLUME30oratory experiments, and observations from fieldstudies. The testing and evaluation prograrn is ongoing andis intimately coupled with further development of themodels and their components. This paper describesresults of a study aimed at evaluating the NUATMOS/CITPUFF configuration of TAPAS using experimentalresults from the CTMD program. In particular, resultsfor Cinder Cone Butte (CCB) (a roughly axisymmetricisolated hill in Idaho) are used to evaluate the performance of the current technique for incorporating terrain and atmospheric stability effects into NUATMOS.A quantitative statistical comparison between predictions and measured tracer concentrations is used, withthe results presented in this paper focusing on the casestudy hours in the modeler's dataset examined by Strimaitis et al. (1983).2. The NUATMOS and CITPUFF modelsa. NUATMOS The model generates a 3D mass-consistent wind fieldbased on arbitrarily located observations. This isachieved by interpolating throughout the domain ofinterest and then making minimal adjustments in orderto eliminate divergence (for mass conservation). The divergence elimination phase minimizes thefunctional~r(u, v, w) = f f f [(u- ~o)2 + (v-- vo)2 + a-2(w - Wo)2[dV ,Jsubject to the constraint Ou Ov Owwhere x, y are the horizontal coordinates; z is the vertical coordinate; V0 = (u0, v0, Wo) is the correspondinginitial (interpolated) velocity; V = (u, v, w) is the finalvelocity; and the parameter a allows horizontal andvertical winds to be adjusted differentially. Mathematically, the functional is minimized usingvariational calculus methods with the resulting Poissonequation for the Lagrange multiplier solved in terrainfollowing coordinates using multiple grids and a tridiagonal solver to speed up convergence. NUATMOS(version 5) has been optimized to the degree that itcan be run comfortably on current-generation personalcomputers. The NUATMOS model has been extensively testedusing comparisons with exact solutions, data from laboratory experiments, and data from field flow measurements (e.g., Ross et al. 1988a,b; Connell 1988).The tests have revealed that for an initially uniformwind the divergence reduction stage with a = 1 willyield the correct potential flow solution for an isolatedterrain shape. The adjustment to the :initial wind fieldcan 'be interpreted as the imposition of a "terrain effect"upon a uniform background wind. This leads tospeedup, retardation, and channeling, as the wind encounters changes in the shape of the underlying terrainsurfi~ce. The potential flow solution can also be regarded as an approximation to flow past obstacles inneutral atmospheric stability conditions. Ross et al. (1988a) use a hybrid approach to developand test a framework for a Froude number dependentexpression for a based on conservation of energy arguments and laboratory data. Subsequent extensionand evaluation using additional laboratory data (Rosset al. 1988b) suggest that the relationship 3 (1) a-a= I +($~_ 1)Fra'where $ is the "speedup" over the terrain and Fr is acharacteristic Froude number of the flow, provides arobust relationship that incorporates the influence ofterrain shape indirectly through the calculated speedup.Testing of this relationship using "direct" flow fielddata; from the CTMD program was inconclusive,largely because of the considerable uncertainties in thelimited data available. The testing of the a-Fr relationship has been restricted to isolated topography embedded in a uniformapproach flow of speed U~o at an upstream referenceheight z = Hs, and a linearly thermally stratified environment characterized by a buoyancy frequency N2= (g/O~)(dOe/dz), where 0e is the environmental potential temperature with 0~ a representative value ofthe air between heights z = H~ and the topographyheight z = H. The hill Froude number Fr = U~/NH,and the speedup $ = Urn,x/U~ is for the potential flowsolution (obtained from NUATMOS with a = 1 ). Inthe present analysis we follow SpangJer (1983, 1986)and use a "release height" Froude number where U~is taken as the release height wind speed and H is thecharacteristic height of the CCB ten:ain (taken to be95 ~n).b. CITPUFF CITPUFF (Ross et al. 1987) is a multisource Gaussian puff model that uses the 3D wind fields generatedby NUATMOS, interpolated in both time and location.Plume rise, stability class, and other hourly model input,'; are also interpolated in time. Other features include a range of optional dispersion formulas, partialpenetration of an elevated stable layer and introductionof an elevated layer of arbitrary stability, and dynamicdetermination of the puff advection :step length. TAPAS, configured to use NUATMOS and CITPUk-F, has been extensively evaluated using data fromthe Latrobe Valley Airshed Study conducted in southeastern Australia (Lorimer 1989; Ross et al. 1989).JULY1991 D.G. ROSS AND D. G. FOX 9113. The Cinder Cone Butte database Cinder Cone Butte is a two-peaked, roughly axisymmetric hill about 100 m high. It's nearly circular baseis about 1 km in diameter. Figure 1 illustrates the terrain (with a vertical exaggeration of a factor of approximately five) with a view looking south, with asection from the topographic file used in the modeling.The experimental program included SF6 tracer releasesupwind of CCB with tracer sampling at receptors located on the hill. A meteorological monitoring networkwas used to obtain direct and derived meteorologicalparameters corresponding to each tracer release experiment. In addition, lidar and photographic dataprovided information on plume trajectories and plumespread. Strimaitis et al. (1983) present the results of casestudy analyses from a total of 45 hourly experimentsconducted. For each case study period the meteorological and tracer gas databases were analyzed in detailto provide a comprehensive modeler's dataset for evaluating the performance of air quality models for stableflows in complex terrain. Each case study dataset includes hourly average values of the observed SF6 concentration at each receptor, source release information,and meteorological characteristics. In the case of themeteorological information, data from a number oflocations were used to derive: - an hourly averaged wind vector (at the source release height h); - atmospheric stability parameters, including theBrunt-Vfiis~l~i frequency N, the critical streamlineheight H~, a characteristic Froude number, and thePasquill stability category; - turbulence data, including the crosswind and vertical turbulent intensities. The case study hours are subdivided according towhether the release height was above or below the critical streamline height. The critical or dividing streamline height (e.g., Sheppard 1956 ) represents a divisionbetween two regions of flow that occur in stable stratified flow around a terrain obstacle. Below the criticalstreamline, the flow has insufficient kinetic energy tosurmount the terrain feature and consequently passes- around it in an approximately horizontal plane. Abovethe critical streamline, the flow passes up and over theterrain feature. It is important to note that the criticalstreamline concept is only a simplistic approximation.In fact, its role in airflow blocking and splitting hasbeen questioned by Smith (1988).4. TAPAS evaluationa. TAPAS configurations Ground-level predictions at the tracer receptor locations have been made for each of the CCB case studyhours using the following configurations of theNUATMOS wind field model: (i) a uniform wind field throughout the computational domain. The value of the wind vector is takenas the source height wind vector (see Table 1 ). Operationally this is achieved by completing only the initialinterpolation phase of NUATMOS. (ii) Adjusting the wind field produced by configuration (i) to be divergence-free by applying the divergence reduction phase of NUATMOS with the parameter a = 1. Ross et al. (1988a) show that this approachwill generate the potential flow solution for any terrainshape. (iii) As for configuration (ii), but with the value ofa specified using Eq. ( 1 ). (iv) As for configuration (ii), but with the value ofa determined empirically by matching the criticalstreamline height predicted by NUATMOS with thatreported in the modeler's dataset for each particularcase study hour. FIG. 1. Ground relief of CCB (exaggerated by a factor of ~5) within the computational gridused by NUATMOS and CITPUFF with a view looking south. The complete terrain file usedconsisted of 61 x 61 points or 60 x 60 cells with a horizontal resolution of 40 m.912 JOURNAL OF APPLIED ]METEOROLOGY VOLUME30 The CITPUFF model was configured to move puffsalong the terrain-following surface passing throughtheir point of release for configuration (i). For the otherconfigurations the CITPUFF model used the fully 3Dwind field (including the vertical velocities) generatedby NUATMOS to transport puffs. The TAPAS configurations have been chosen tohighlight the influence of the technique for incorporating terrain and atmospheric stability effects in theNUATMOS wind-field model. We axe interested in,first, the addition of a 3D wind field and, second, thequality (mathematical and physical correctness) of thefield on the concentration predictions in complex terrain. Predictions from the CCB models (Strimaitis et al.1983) for h > Hc: NEUTRAL and LIFT, and for h~< Hc: IMPINGEMENT and WRAP, :are also includedin the analysis for comparison with tile TAPAS configurations. It is relevant to note that the LIFT and WRAPmodels were developed as improvements to the NEUTRAL and IMPINGEMENT models, respectively. TheCCB models are an earlier form of the CTDM (Strimaitis et al. 1987) and CTDMPLUS (Perry et al. 1989)modeling systems. In what follows, the models or system configurationswill be referred to by the following code:CCB-N/I: NEUTRAL (h > Hc) and/or IM PINGEMENT ( h ~< Hc)CCB-L/W: LIFT (h > Hc) and/or 'WRAP (h ~< He)TAP-UNI: T^P^S [NUATMOS configuration (i), uniform wind field]TAP-POT: T^P^S [NUATMOS configuration (ii), potential flow field]TAP-aF: TAP^S [NUATMOS configuration (fii), a determined using equation ( 1 ) for mulation ]TAP-aE: T^P^S [NUATMOS configuration (iv), a determined empirically]. The basic output from a deterministic air qualitymodel is an estimate of the ensemble average concentration at a receptor point. The uncertainty associatedwith this prediction is made up of a reducible component due to inadequacies of the model formulationand input data, and an irreducible component due tothe random nature of turbulence in the atmosphere(Fox 1984). The basic input requirements for the CITPUFF model are the emission rate, release height, dispersion relations, gridded terrain heights, and the flowfield provided by NUATMOS. The CCB database enables us to minimize the uncertainties associated withthe first four of these inputs and to focus attention onthe influence of the flow field, and iin particular theeffects of terrain and atmospheric stability on the flow.The uncertainties associated with emission rate, releaseheight, and terrain heights are deafly small. We adoptthe dispersion relations used in the CCB models fly = iy.X, izX ( 1 + Nxl U~p)where ay and 0'~ are the crosswind and vertical standarddeviations of the assumed Gaussian distribution, respectively; iy and i~ are the crosswind and vertical turbulence intensity (at source height), respectively;is the wind speed at source height; x is travel distancefrom the source; and p is a parameter determined bycalibrating against the lidar measure, ment (p = 1.5).Although the dispersion relations for CCB are basedon the experimental data, they are still subject to considerable uncertainty (Strimaitis et al. 1983). We also examine the effect of usiing the PasquillGiflbrd dispersion scheme, but with the standardscheme adjusted for roughness height and averagingtime in the manner recommended by Hanna et al.(1977). Table 1 summarizes the basic input data used forthe model simulations for each of the case study hoursexamined in this analysis.b. Evaluation procedure We adopt the approach of Hanna (1989) and usetwo. basic statistical performance measures to assessthe performance of the alternative models or systemconfigurations. The first measure is the normalized orfractional bias of the mean concentration: FB = 2(~o - Z~)/(Z0 + ~),where Z0 and ~, are the mean observed and predictedconcentrations, respectively. The fractional bias FB hasan ]ideal value of zero and can vary between -2 and+2. The second performance measure is the normalized mean-square error NMSE: NMSE = ( c--~-~-~)2/ZoZ~,which provides a global error estimate of the scatterbetween observations and predictions. Means and confidence intervals of the two performance measures are calculated using the resamplingand model evaluation software described by Hanna(1989). The so-called bootstrapping resampling procedare has been used to estimate the confidence limitsor statistical uncertainty of the performance measureresults. The estimates of 95% confidence limits havebeen obtained using both the "seductive" and "moment' bootstrap procedures, althougJa only the resultsof the latter are presented here. Hanna suggests thatthe latter procedure is preferable as it overcomes problems that can arise with the former at the tails of thedistribution. For the present case the differences in estimates obtained from the alternative procedures arenegligible and do not alter our conclusions.JULY 1991 D.G. ROSS AND D. G. FOX 913 TABLE 1. Observed and derived input data for the model simulations. Derived modelMeteorological dataa Source dataa inputsU,o (h) 0 N Stability h Q Distance DirectionExperiment (m s-l) (deg) He iz iy (s-l) class (m) (g s-l) (m) (deg) Frb a- ad202H4 7.3 327 0 0.041 0.054 0.037 E 20 0.082 1014.6 319 2.08 0.60 0.26202H5 7.8 326 7 0.037 0.060 0.047 E 30 0.086 1014.6 319 1.75 0.53 0.25204H1 2.1 359 58 0.049 0.341 0.051 F 30 0.094 1035.7 5 0.43 0.15 0.10205H4 5.9 121 10 0.031 0.069 0.042 F 40 0.083 t155.1 120 1.48 0.47 0.24205H5 6.5 120 24 0.049 0.104 0.048 F 50 0.087 1155.1 120 1.43 0.46 0.19206H4 4.5 128 20 0.031 0.078 0.044 F 35 0.039 595.9 124 1.08 0.36 0.21206H6 2.1 127 35 0.086 0.212 0.035 F 35 0.062 595.9 124 0.63 0.22 0.17206H8 2.0 127 37 0.097 0.180 0.040 F 35 0.062 595.9 124 0.53 0.19 0.16209H7 2.5 105 58 0.047 0.121 0.094 F 40 0.160 999.2 101 0.28 0.10 0.10210H3 6.3 118 26 0.018 0.105 0.044 F 57 0.169 1084.3 114 1.51 0.48 0.19210H7 7.3 118 14 0.057 0.108 0.035 F 58 0.178 1086.2 122 2.20 0.62 0.22211HI 1.9 107 45 0.084 0.326 0.058 F 30 0.179 1001.2 101 0.34 0.12 0.14211H5 2.4 120 26 0.068 0.277 0.092 F 20 0.175 1155.1 120 0.27 0.10 0.19 a Meteorological and source data from Strimatis et al. (1983). Hourly average meteorological data--U~ (h): wind speed at release height(m s-l); 0: wind direction (deg) (0 = 0, northerly wind); He: dividing streamline height (m); iy, iz: crosswind and vertical turbulence intensities;N: Brunt-V/iis~l~i frequency (1 s-l); Pasquill stability class (Lavery et al. 1982; Table 42). Source Data--h: release height (relative to terrainheight datum) (m); Q: hourly average emission rate of SF6 (g s-t); Source position relative to "center" of CCB grid (see Fig. 1). b Release height Froude number Fr = Uo~/NH, where H = 95 m is taken as the characteristic terrain height of CCB. c a values determined from Eq. (1) using the release height Froude number and a "neutral" flow (a = 1) speedup ofS = 1.18. d a values determined by matching the experimental and simulated value of Scatterplots of observed versus predicted concentrations are also used as a simple depiction of the model'sresults. The evaluation procedure is applied to an ensembleof 13 case study hours with the data combined andblocked; that is, the observation/prediction pairs ateach receptor for all hours are combined into a singledataset (455 observation/prediction pairs), which isdivided into blocks containing data with similar characteristics. The justification for combining and examining all the case study hours as a single dataset isbased on the fact that all l 3 hours represent stable atmospheric conditions (Pasquill stability categories Eor F). The obvious blocking is then to divide the databased on whether h > Hc or h ~< He. With the blocked bootstrap resampling procedure,each resample of 455 observation/prediction pairs isforced to contain 233 pairs (equivalent to the 7 casestudy hours where h > He) from the first block and222 pairs (equivalent to the 6 case study hours whereh ~< He) from the second block. Consequently, it isimpossible to pick all 455 pairs from the same blockin a given sample. The evaluation procedure is also applied separatelyto the two component ensembles where h > H~ and h~<Hc.5. Results A simple but instructive way of illustrating the resultsis through a scatterplot of predictions versus observations, paired in location and time. Figures 2a and bshow such plots for each model configuration, but withthe results broken into ensembles of hours for whichh > He and h ~< Hc, respectively. For the ensemblewith h > He, the improvement in predictive performance resulting from including each of the wind fieldadjustments associated with moving from TAP-UNIthrough to TAP-aE is clear, with the TAP-aE configuration being the best performer. The ensemble withh ~< Hc also indicates some progressive improvement;however, the predictive performance of all model configurations is relatively poor. In particular, each configuration badly underpredicts the highest observedconcentrations, as well as tending to overpredict thelow end of the range. In fact, there is evidence that theinclusion of each wind-field adjustment associated withmoving through TAP-UNI to TAP-aE leads to a deterioration in predictive performance for the lower observed concentrations. It is useful to be able to quantify these observations,and to have estimates of the statistical confidence ofthe findings. Table 2 contains the calculated values of FB andNMSE for the ensemble of all hours and the two component ensembles, for each model or system configuration. Figures 3 and 4 illustrate FB and NMSE and their95% confidence intervals for the total and each of thecomponent ensembles. If the confidence limits overlapthe zero of a statistical measure, then the calculatedvalue of the measure is not significantly different from914 JOURNAL OF APPLIED METEOROLOGY VOLUME30zz a. CCB-L/W /~'.0I.~o.~ ':'-o o'.~ do ,:~ ~;~ i.~ IObserved ppt(xlO'2) Observed ppt (x I0-2)zo, a. TAP-UNI , //1.5'0.5' 0 0.~ LO L~ ~.0 ~. Observed ppt xlO'~1Observed ppt (xlO-2) ., ,0 0. 1.0 1.5 2.0 2.5Observed ppt(xlO-2)Observed ppt (xlO-2) FiG. 2. Scatterplots of predicted versus observed concentrations (ppt), paired in location and time, for e, ach modelor system configuration (a) h > Hc and (b) h ~< He. The area between the dashed lines contains points that are withina factor of 2 of the solid (perfect fit) line.JULY 1991 D.G. ROSS AND D. G. FOX 915Observed ppt (xlO-$)]2~ 2.0,1.5,1.0 b. CCI~L/W / + +, +/ + ,/' ,'F 4- / ~ .. / + ..-/' ' ,i,;~~~~ ~""" ........... ~..~_.~' + .-~-.'-' ,0 ~-~ ~' + o ~ ~o ~ ~o ~',~ Observed ppt (xlO-~)2~'b.2.0. ///'0.5. 0 0.5 1.0 1.5 2.0 2.5Observed ppt(xlO-~)1 2.~] b. TAP-POT/ /I .$ "/' 1 ,., / I '~:~+ +~+ + + + 0 ~ ,, ~ ~ ,~ ~ , , , 0 0.5 1.0 1.5 2.0 2.5Observed ppt(xlO-~Observed ppt(xlO-~)Observed ppt(xlO-5)FIG. 2. (Continued)916 JOURNAL OF APPLIED iMETEOROLOGY VOLUME30 TABLE 2. Statistical perforrnance measures. Fractional bias (FB) ModelEnsemble CCB-N/I CCI:I-L/W TAP-UNI TAP-POT TAP-aF TAP-aE Total 0.26 -0.003 0.97 0.72 -0.04 -0.16h > Hc -0.06 0.17 1.27 0.80 0.37 -0.004h < He 0.37 -0.05 0.89 0.69 -0.14 -0.2 Normalized mean-square error (NMSE) ModelEnsemble CCB-N/I CCB-L/W TAP-UNI TAP-POT TAP-aF TAP-aE Total 4.6 3.9 9.3 6.3 3.7 3.3h > Hc 1.6 2.4 10.8 4.3 1.9 1.4h ~< Hc 4.0 2.9 6.3 4.6 2.6 2.40.0 at the 95% confidence level. Figure 4 shows thatthe NMSE for all models is significantly different fromzero at the 95% confidence level. For the total ensemblewe see from Fig. 3 that the FBs for CCB-N/I, TAPUNI, and TAP-POT are also significantly differentfrom zero, with each model underpredicting. The FBsfor CCB-L/W, TAP-aF, and TAP--aE are not significantly different from zero at the 95% confidence FB 1.4 1.2 1.0' 0.8' 0.6' 0.4' 0.2' 0-0.2'-0.4 Icc~-N/] T I I &.Icc~-[/wTAP-POTTII: TT TAP-~E F~G. 3. Ninety-five percent confidence intervals on FB for eachmodel or system configuration as estimated by Hanna's (1989) moment bootstrapping resampling procedure: - total ensemble (solid line) - h > He ensemble (dashed line) - h < He ensemble (dotted line).If the confidence interval does not overlap 0.0, then we have 95%confidence that the FB is different from 0.0.level. The same finding also holds for each componentensemble for TAP-UNI, TAP-POT, and TAP-aE andfor the component ensemble with h ~< Hc for TAPaF, while TAP-aF underpredicts for the h > Hc ensemble. When examining the CCB models, it is importantto remember that each total ensemble comprises predictions from distinct models whose design and rangeof applicability are restricted to either h > Hc or h ~< Hc.The total ensemble has some meaning and usefulnesswhen comparing with the TAPAS configurations, butshould not be examined in comparison to the resultsNMSE1412I08 6 4_IT 2 cc~w[ 0 TAP-UNI ~.~I- T^P-P-~ IIIi I? xCCB-L/W TAP--=F I TAP-~E FIG. 4. Ninety-five percent confidence intervals on NMSE for eachmodel or system configuration as estimated by Hanna's (1989) moment bootstrapping resampling procedure: - total ensemble (solid line) - h > He ensemble (dashed line) - h ~< H, ensemble (dotted line).If the confidence interval does not overlap 0.0, then we have 95%confidence that the NMSE is different from 0.0.JULY 1991 D.G. ROSS AND D. G. FOX 917918 JOURNAL OF APPLIED METEOROLOGY VOLUME30OJm 41 II I4'! II II, i '7 TJULY 1991 D.G. ROSS AND D. G. FOX 919for its component ensembles. For the ensemble withh > He, the NEUTRAL model (CCB-N) yields a zeroFB while its supposed improvement, the LIFT model(CCB-L), underpredicts. For the case h ~< He, however,the newer WRAP model (CCB-W) has a zero FB,while the older IMPINGEMENT model (CCB-I) underpredicts. We now examine the influence of the relative performance of the model or system configurations byusing the resampling procedure to estimate confidencelimits on differences in FB (AFB) and NMSE(ANMSE) between models. The results of all possiblepairings of the models are illustrated in Figs. 5 and 6for each of the three ensembles. We consider first the influence of the flow field adjustments contained in the alternative NUATMOSconfigurations. The confidence intervals on both AFBand ANMSE confirm our conclusions from examiningthe scatterplots and show that there is a significant improvement in predictive performance as we move fromTAP-UNI through to TAP-aE. This conclusion isvalid also for the cases with h > Hc and h ~< He. However, it should be noted that the differences betweenTAP-aF and TAP- aE are at the margin of significancefor both the total ensemble and the component ensemble with h ~< He. The improvement in predictive performance in moving from TAP- aF to TAP- aE is clearfor the cases where h > He.FB CCB-N/I-0.2'-0.4-0.5.T TTTAP--UNI T TAP-POTCCB-L~V ITAP~E FIG. 7. Ninety-five percent confidence intervals on FB for eachmodel or system configuration when CCB dispersion scheme is replaced by Pasquill-Gifford scheme in the TAPAS configurations: - h > He ensemble (dashed line) - h < Hc ensemble (dotted line).If the confidence interval does not overlap 0.0, then we have 95%confidence that the FB is different from 0.0.NMSE 14 12' I0' 8' 6' T 4 .L 2' CCB-N/I O T ~ .- TAP-UNICCB-L/W TAP-POTIi ~-"TAP-.F TAP~E FIG. 8. Ninety-five percent confidence intervals on NMSE for eachmodel or system configuration when CCB dispersion scheme is replaced by Pasquill-Gifford scheme in the TAPAS configurations:- h > Hc ensemble (dashed line)- h < He ensemble (dotted line).If the confidence interval does not overlap 0.0, then we have 95%confidence that the NMSE is different from 0.0. For the ensemble h > He, the NEUTRAL (CCBN) and LIFT (CCB-L) models are significantly different with regard to both NMSE and FB, with the olderNEUTRAL model performing significantly better thanits "improvement." The NMSE and FB for the IMPINGEMENT (CCB-I) and WRAP (CCB-W) modelsare also significantly different for the case h ~< Hc; however, now the improved model WRAP is a significantlybetter performer than its predecessor IMPINGEMENTwith respect to both statistical measures. Let us now compare the best of the TAPAS configurations with the best of the CCB configurations. Forthe total ensemble, the performances of CCB-L/Wand TAP-aE are not significantly different with respectto either NMSE or FB. This is also true for the casewhere h ~< He. However, for the ensemble with h > He,the TAPAS configuration TAP-aE has a significantlysmaller NMSE than the NEUTRAL model (CCB-N),with both models having no significant difference intheir FB. Indeed, we saw earlier that the FB for bothmodels is not significantly different from zero (Fig. 3 ). Figures 7-10 illustrate the results of repeating theanalysis when the CCB dispersion scheme is replacedby the Pasquill-Gifford scheme in the TAPAS configurations. Only the results for the ensembles with h > Hcand h ~< Hc are presented. The results confirm our conclusion that the inclusionof the flow field adjustments in the alternative NUATMOS configurations do yield a significant improvementin predictive performance, although the choice of thebest TAPAS configuration is less clear than for the CCBdispersion scheme.920 JOURNAL OF APPLIED METEOROLOGY VOLUME30 AFB 1,61.4.1.21,00.80.60.40.20.0'-0.2-0.4-0.6-0.8-I .0'-I .2-I.4I-6I-2 I 2-4 I-4 !~5II(a)4-64-5 T5-6 AF.'B 1.4,1.2'.1.00.80.6'0.40.20.0-0.2-0.4- 0.6-0.8-I .C)-I .;!-I.4I-2 I-3 I II-4(b) FIG. 9. Ninety-five percent confidence intervals on differences in FB (AFB) between models or system configurations when CCBdispersion scheme is replaced by Pasquill-Gifford scheme in the T^P^S configurations ( 1 = CCB-N/I, 2 = CCB-L/W, 3 = TAPUNI, 4 = TAP-POT, 5 = TAP-aF, and 6 = TAP-aE) (a) h > Hc ensemble and (b) h ~ Hc ensemble. If the confidence intervaldoes not overlap 0.0, then we have 95% confidence that the FBs for the two models are different. For the ensemble with h > Hc, both TAP-aF andTAP-aE are not significantly different with respect toNMSE; however, TAP-aF slightly underpredicts whileTAP-aE tends to slightly overpredict. As a consequence, both models have a slightly inferior predictiveperformance compared to the best of the CCB models(CCB-N), which have a FB that is not significantlydifferent from zero. The NMSE values of all threemodels (TAP-aF, TAP-aE, and CCR-N) are not significantly different. For the ensemble with h ~< He, TAP-aF is the bestof the TAPAS configurations with a significantly smallerNMSE than TAP-aE, although both models have aFB that is not significantly different fi:om zero. Comparison with the best of the CCB models (CCB-W)indicates that the TAPAS configuration may be superioron the basis of its significantly smaller NMSE, althoughboth models (TAP-aF and CCB-W) have a FB thatis not significantly different from zero. Figures 1 1 a and b contain scatterplots of predictionsversus observations for the best of tlhe TAPAS configurations based on the Pasquill-Gifford scheme for theensemble of hours for which h > H,, and h ~< Hc, respectively. Comparison with the scatterplots for theappropriate CCB models in Fig. 2 provides a simpleillustration of the results discussed previously.6. Discussion and conclusions A principal objective of this work is the evaluationof the current technique for incorporating terrain andatmospheric stability effects into the 3D diagnosticwind-field model NUATMOS when only a backgroundwind vector is available. Although based on the use ofJULY 1991 /~NMSE 12ROSS AND D. G. ~XNMSE 12FOX921I08'64 2 I-5 I-6 o T T T I I-2 I-4 -2'-6- ~'~ I-8-113-12 3-5 3-6iii2-32-5 2-6 4-5 4-6I I I II0'8'6.4 I- I-6 I'fI 0 2-3 -10(a) -12. (b) 3-5 3-62-5 T I 4-5 4-6i z~6 ~-~ T 'r 5-6 ' F1G. 10. Ninety-five percent confidence intervals on differences in NMSE (^NMSE) between models or system configurationswhen CCB dispersion scheme is replaced by Pasquill-Gifford scheme in the T^P^S configurations ( 1 = CCB-N/I, 2 = CCB-L/W,3 = TAP-UNI, 4 = TAP-POT, 5 = TAP-aF, and 6 = TAP-aE) (a) h > Hc ensemble and (b) h ~< Hc ensemble. If the confidenceinterval does not overlap 0.0, then we have 95% confidence that the FBs for the two models are different.2,52.0I.,51.0Observed ppt (x I0'2)Observed ppt (x I0'3) FIG. 11. Scatterplots of predicted versus observed concentrations (in ppt ), paired in location and time, for the "best"of the TAP^S configurations TAP-aF based on the Pasquill--Gifford dispersion scheme (a) h > He and (b) h ~< He.The area between the dashed lines contains points that are within a factor of 2 of the solid (perfect fit ) line.922 JOURNAL OF APPLIED METEOROLOGY VOLUME30"indirect" field measurements, the statistical analysisadopted, together with the form of the CCB data, enables us to focus on these key flow fie, ld adjustments. The results presented for the ensemble of case studyhours and for two component ensembles; one withsource release heights above the cfi.tical streamlineheight and the others below this height, demonstratethat significant improvement in the predictive abilityof T^P^S results when the initially uniform flow field(based on interpolation of the background wind) ismade divergence-free (with a = 1 ) in response to theterrain shape. The improvement in performance isconsiderably larger for the case where h > Hc than forthat where h ~ Hc. The predictive performance of T^P^S is seen to befurther improved when the parameter a is used to incorporate atmospheric stability effects during the divergence reduction stage of NUATMOS. Determininga on the basis of a characteristic Froude number appears to be slightly inferior to using a value obtainedby matching the experimental and predicted criticalstreamline heights, although this finding is at the margin of significance (at the 95% confidence level) andis reversed for the cases where h ~ H,~. when the CCBdispersion scheme is replaced by the Pasquill-Giffordscheme. The ensemble of cases where h > He is inconclusive. A secondary objective was to evaluate the overallperformance of TAPAS and compare its performancewith that of models specifically developed from theCCB dataset. Comparison between the best of the T^P^S and CCBconfigurations indicates that TAPAS perforrfls as wellbut not better than the CCB models,, except for theensemble of cases where h > Hc when TAPAS yields asignificantly lower NMSE. This result is encouraginggiven that the CCB models have been. developed andcalibrated using the CCB dataset itself; whereas, apartfrom the CCB dispersion scheme, T^PAS uses basicinput information and subsequently derived flow fieldadjustments. The results obtained by replacing the CCB dispersionscheme with the Pasquill-Gifford scheme are evenmore encouraging given that the TAPAS results are nowalmost totally based on inputs that are either routinelyavailable or readily derived from routinely availableinformation. We have not attempted to compare the relative performance of the two dispersion schemes used in TAPASor to conduct a "scientific evaluation" of them. Thedispersion scheme and its interaction with the flow fieldrepresents a weak link in most models and, apart fromthe inherent uncertainties associated wi'th the stochasticatmosphere, is a major cause of the relatively largeNMSE resulting from even the best of the models examined here. As air flows past CCB, the terrain-induced flow fieldwill produce kinematic effects on the spread and growthof a plume. For example, the speedup over the crestwill result in streamline compression in the vertical,with the streamlines also approachintg the terrain surface more closely. In contrast, the streamlines in thecrosswind direction will "fan out" over the crest. Thesekinematic influences will act to decrease az and increaseay over the crest (e.g., Egan 1975). However, it is important to realize that final plume a,r puff spread depends also on the turbulent diffusion rates that resultfrom these kinematic effects. Hunt and Mulhearn(19'73), in examining the theory of turbulent plumesimbedded within potential flow fields, point out thatturbulent diffusion across streamline, s is enhanced bycontraction in the distance between streamlines in thevertical and retarded by the expansion in the distancebetween streamlines in the lateral. The TAPAS system, via NUATMOS, simulates theterrain-induced effects on the strearnlines in terms oftheir relative separation and closeness to the terrain,with the parameters a playing a major role in controll[ing the magnitude of these influences. However,the CITPUFF dispersion algorithm and its couplingwith the flow field does not directly include the fullkinematic and turbulent diffusion rate influences onthe spread of puffs. In particular, each puff is transported by only the velocity at its centroid, and its dispersion about the centroid is effectively independentof the flow field. There is, of course, an indirect influence of the flow field on the "effective" puff dispersionused in calculating the ground-level concentration fieldvia the puff size at a particular location and the positionof its centroid relative to the terrain. The results presented are part of an ongoing studyto evaluate and further develop TAPAS and its component models using experimental results from theCTMD program. The Hogback Ridge dataset will enable us to focus on an isolated, approximately 2D ridge,while the Tracy power plant dataset will enable us toevaluate the findings from the simple: isolated 2D and3D terrain in the context of morn complex terrain. One advantage of the analysis presented in this paperis that it provides confidence in the relative applicabilityof rnodels for use in regulatory settings. Although thedataset is not "realistic" from a regulatory perspective,it does allow specific presentation of over- and underprediction. Inasmuch as responsible regulatory acceptance of models requires knowledge of and displayof these characteristics, the TAPAS configurations presented here have regulatory utility. Acknowledgments. The support provided under Cooperative Agreement 28-C8-478 between MonashUniversity and the Rocky Mountain Forest and RangeExperiment Station, USDA, is gratefully acknowledged. The assistance of Mr. Andrew Lewis in performingthe model runs and developing many of the graphicsand analysis routines is also gratefully acknowledged.JULY 1991 D.G. ROSS AND D. G. FOX 923REFERENCESConnell, B. H., 1988: Evaluation of a three-dimensional diagnostic wind model: NUATMOS. Unpublished M.S. thesis, Department of Natural Resources, Colorado State University, Fort Collins, 135 pp.DiCristofaro, D. C., D. G. Strimaitis, B. R. Greene, R. J. Yamartino, A. Venkatram, D. A, Godden, T. F. Lavery and B. S. Egan, 1985: Environmental Protection Agency complex terrain model development program: Fifth Milestone Rep.-1985. EPA-600/ 3-85-069, United States Environmental Protection Agency, Re search Triangle Park, NC.Egan, B. A., 1975: Turbulent diffusion in complex terrain. Lectures in Air Pollution and Environment Impact Analysis, D. A. Hauger, Ed. Amer. Meteor. Soc.Fox, D. G., 1984: Uncertainty in air quality modeling. Bull. Amer. Meteor. Soc., 65, 27-36. -, D. G. Ross, D. L. Deitrich and D. E. Mussard, 1987: An update on T^~'^S and its model components. Preprint Volume, Ninth Conference on Fire and Forest Meteorology, San Diego, Amer. Meteor. Soc., 135-137.Hanna, S. R., 1989: Confidence limits for air quality model evalu ations, as estimated by bootstrap and jackknife resampling methods. Atmos. Environ., 23, 1385-1398. , G. A. Briggs, J. Deardorff, B. A. Egan, F. A. Gifford and F. Pasquill, 1977: Summary of recommendations made by the AMS workshop on stability classification schemes and sigma curves. Bull. Amer. Meteor. Soc., 58, 1305-1309.Hunt, J. C. R., and P. J. Mulhearn, 1973: Turbulent dispersion from sources near two-dimensional obstacles. J. Fluid Mech., 61, 245 274.Lorimer, G. S., i 989: Validation of air pollution dispersion models.Clean Air (Aust.), 23, 82-88.Perry, S. G., D. J. Burns, L. A. Adams, R. J. Paine, M. G. Dennis, M. T. Mills, D. G. Strimaitis, R. J. Yamartino and E. M. Insley, 1989: User's guide to the complex terrain dispersion model plus algorithms for unstable situations (CTDMPLUS) Vol. 1: Model description and user instructions. Environmental Protection Agency Rep. EPA / 600 / 8-89/041, United States Environmental Protection Agency, RTP, NC, 196 pp.Ross, D. G., G. S. Lorimer, L. Li and I. N. Smith, 1987: "CITPUFF": A Gaussian Puff Model for estimating pollutant concentration in complex terrain CAMM Rep. No. 21/87, 72 pp. [Available from Chisholm Institute of Technology: P.O. Box 197, Caulfield, Victoria, Australia.]--, I. N. Smith, P. C. Manins and D. G. Fox, 1988a: Diagnostic wind field modeling for complex terrain: Model development and testing. J. Appl. Meteor., 27, 785-796.--, M. Krautschneider, I. N. Smith and G. S. Lorimer, 1988b: Diagnostic wind field modeling: Development and validation. End of Grant Rep. to Department of Resources and Energy, National Energy Research Development, and Demonstration Program. End of Grant Rep. No. NERDDP EG89/776, 108 PP.--, G. S. Lorimer, D. G. Fox and I. N. Smith, 1989: Results of applying wind and dispersion models in complex topography. Preprint Volume, 82nd Annual Meeting and Exhibition. Air and Waste Management Association, Anaheim, 1-16.Sheppard, P. A., 1956: Airflow over mountains. Quart. J. Roy. Meteor.Soc., 82, 528-529. Smith, R. B., t988: Linear theory of stratified flow past an isolated mountain in isosteric coordinates. J. Atmos. Sci., 45, 3889 3896.Smith, I. N., and D. G. Ross, 1988: Diagnostic wind field studies. Clean Air (Aust.), 22, 141-143.Spangle2, T. C., 1983: Stagnation zone impaction on a simple terrain feature in stable flow. Ph.D. dissertation, Utah State University, Logan, Utah, 175 pp.--., 1986: The role of near terrain turbulence in the prediction of ground level pollutant concentrations in complex terrain. Atmos. Environ., 20, 861-865.Strimaitis, D. G., A. Venkatram, B. R. Greene, S. Hanna, S. Heisler, T. F. Lavery, A. Bass and B. A. Egan, 1983: Environmental Protection Agency complex terrain model development pro gram: Second milestone rep.--1982. EPA-600/3-83-015, United States, 375 pp.--., R. J. Paine, B. A. Egan and R. J. Yamartino, 1987: Environ mental Protection Agency complex terrain model development: final rep. Environmental Protection Agency report EPA/600/ 3-88/006, Environmental Protection Agency, RTP, NC, 486 pp.