## 1. Introduction

Computational fluid dynamics (CFD) codes are used at the microscale to predict atmospheric boundary layer flows over complex terrain for a variety of applications, ranging from the siting of wind turbines to predictions of flow in urban terrain for contaminant dispersion. CFD codes used for simulations of the atmospheric boundary layer often lack important features, such as incorporating atmospheric physics and regional weather effects. Additionally, it is often time consuming to create a high-quality computational mesh for simulations of complex geometries (terrain) because of the manual grid manipulation that can be required. Mesoscale numerical weather prediction models are being increasingly used at higher resolutions, tending toward the microscale where CFD codes are traditionally used. At these smaller scales, mesoscale models face several inherent limitations that prevent accurate simulations in complex terrain. One limitation is the use of terrain-following coordinates in most mesoscale models. This coordinate accommodates complex terrain by transforming the physical domain onto a Cartesian grid, thereby simplifying the application of lower boundary conditions. The transformation introduces metric terms into the governing equations, which when discretized, introduce truncation errors. These coordinate transformation errors significantly degrade the quality of the solution in steep terrain, as demonstrated by previous researchers (Janjić 1977; Mahrer 1984; Schär et al. 2002; Klemp et al. 2003; Zängl 2002, 2003; Zängl et al. 2004; Lundquist et al. 2010b).

To improve the prediction of atmospheric flows in complex terrain, we have implemented an immersed boundary method (IBM) in the mesoscale Weather Research and Forecasting (WRF) model, referred to in this paper as IBM-WRF. IBM is a gridding technique that allows complex terrain to be modeled without a coordinate transformation, therefore eliminating restrictions on terrain slope and the errors associated with coordinate mapping (Iaccarino and Verzicco 2003; Mittal and Iaccarino 2005). Previous researchers have implemented similar Cartesian or nonconforming methods in mesoscale models. Trini-Castelli and Reisin (2010) use a Cartesian grid with a masking method in the Regional Atmospheric Modeling System (RAMS) to simulate flow around rectangular buildings. With the masking method the building boundaries align with the Cartesian grid, creating a stair step representation of the terrain. The shaved cell method has been successfully used in models based on the finite-volume method (Adcroft et al. 1997; Walko and Avissar 2008; Lock et al. 2012). The shaved cell method cannot be directly applied to the WRF model without modification to the finite-differencing discretization, and tracking the cut-cell interfaces would be complicated by the time-varying pressure-based coordinate.

With our IBM, coordinate surfaces pass through the terrain, and boundary conditions are applied within the interior of the computational domain through the addition of a forcing term in the governing equations. Our IBM creates a piecewise linear representation of the terrain, and handles both Dirichlet and Neumann boundary conditions. The velocity boundary condition used here is no-slip, which is commonly applied in simulations of flow in urban areas. Preliminary work on the addition of a surface drag parameterization appropriate for general atmospheric applications can be found in Lundquist (2010). Additionally, realistic surface forcing can be provided at the immersed boundary by atmospheric physics parameterizations, which are modified to include the effects of the immersed terrain. A version of our IBM suitable for two-dimensional terrain was presented in Lundquist et al. (2010a). In this work, we extend the method to accommodate fully three-dimensional terrain through modification of the IBM algorithm and the introduction of new interpolation methods. IBM-WRF is also now capable of running on highly parallelized machine architectures.

A key component of the immersed boundary method is the formulation of the forcing term used to impose the correct boundary condition at the immersed surface. Because the immersed surface is not necessarily coincident with the grid, an integral part of the immersed boundary algorithm is the interpolation method used in the calculation of the forcing term. In this work, we examine the use of two different interpolation methods that are described in section 2. In the first interpolation method, which is trilinear, weighting coefficients are determined by inverting a Vandermonde matrix. This method is chosen here because one or multidimensional linear interpolation methods were the first interpolation methods to appear in the IBM literature, and are still commonly used (Tseng and Ferziger 2003; Balaras 2004; Ghias et al. 2007; Mittal et al. 2008). In the second interpolation method developed by Franke (1982), weighting coefficients are determined as a function of inverse radial distance. The inverse distance weighted (IDW) interpolation method provides more flexibility in the choice of points influencing the interpolation because the Vandermonde matrix is eliminated, thereby eliminating the constraint that the matrix be well conditioned. Both interpolation methods presented here are uniquely developed to have additional favorable properties beyond those previously used by other researchers. These methods are developed to work with our IBM-WRF implementation, but are general and can be used in other CFD codes. Verification simulations for each interpolation method are presented in section 3 for flow over a three-dimensional hill, and the results are compared to those using terrain-following coordinates.

While both interpolation methods produce accurate results for the three-dimensional hill, only the algorithm using the inverse distance weighted interpolation method is robust enough to handle the highly complex urban terrain found in downtown Oklahoma City, Oklahoma. The accuracy of the inverse distance weighted IBM for simulation of flow and scalar transport in urban terrain is examined in section 4 by simulating intensive observation period 3 (IOP3) of the Joint Urban 2003 field campaign. The urban terrain case uses one-way nesting to provide turbulent flow from a neutrally stratified atmospheric boundary layer at the lateral boundaries of the urban domain. This case is semi-idealized as it does not include mesoscale weather effects or atmospheric physics. Additional changes were made to enable simulation of the IOP3 scalar release including adding the functionality for continuous scalar releases to the WRF code, as well as modifying the boundary conditions on the inner urban nest to allow the scalar release to use open boundary conditions, and therefore pass out of the domain, while nested boundary conditions are imposed on other prognostic variables, such as velocity. Conclusions and future work are discussed in section 5.

## 2. Numerical method

### a. Background on the immersed boundary method

**F**

_{B}in the conservation equations for momentum and scalars

*ϕ*[Eq. (1)]:

**V**is the velocity vector,

*α*is specific volume,

*ν*is viscosity,

*g*is gravity, and

*F*is additional scalar forcing. The body force term takes a zero value away from the boundaries, but modifies the governing equations in the vicinity of the boundary. Generally, IBMs include a determination of the forcing term, and an interpolation scheme to reconstruct the boundary condition on the immersed surface, which is not coincident with computational nodes. In this work, a method commonly referred to as direct or discrete forcing is used (Mohd-Yusof 1997). With direct forcing, the velocity or scalar value is modified at forcing points near the boundary to enforce the boundary condition, eliminating the need for explicit calculation of the body force term in the numerical algorithm. Within the direct forcing class of methods, we have adopted an approach where forcing is applied at ghost cells, defined as the layer of computational nodes located just within the solid domain.

_{ϕ}For each interpolation method, the location of the ghost cell is reflected across the boundary, as in Fig. 1, and this location is labeled as an image point. Next, one of the interpolation methods is applied to determine the value of the variable at the image point. Neighbors for the interpolation differ between the two methods, but can include both computational nodes and points on the boundary. The algorithm for choosing the neighbors is described further in sections 2b(1) and 2b(2), but in general, neighbors are determined by proximity to the image point.

### b. Three-dimensional interpolation methods

We describe two interpolation schemes here: trilinear and inverse distance weighted interpolations. The methods presented in this work are unique from those used by other researchers. Our three-dimensional trilinear interpolation method shares the benefits of the two-dimensional method described in Lundquist et al. (2010a). Mainly, our method eliminates the occurrence of numerical instabilities, does not require the use of an iterative solver, and works with the moving pressure coordinates used in WRF. Our inverse distance weighted method requires fewer applications of the interpolation method than the IBM described in Gao et al. (2007). Furthermore, Gao et al. (2007) only addressed the use of Dirichlet boundary conditions, while we additionally address the use of Neumann boundary conditions.

#### 1) Trilinear interpolation

**c**in the interpolant are determined by solving a linear system of equations

**c**=

**for each ghost point, where the rank is equal to the number of neighbors.**

*ϕ***are dependent on the neighbors chosen for the interpolation and the type of boundary condition being imposed. For Dirichlet boundary conditions, Eq. (4) appears in the matrix equation. If the neighbor is a computational node, then**

*ϕ***takes the value calculated at the node. If the neighbor is a boundary point, then the boundary condition is assigned to**

*ϕ***. For Neumann boundary conditions, the gradient of the interpolation function is substituted into the boundary condition, and Eq. (5) results, where**

*ϕ***n**= (

*n*,

_{x}*n*,

_{y}*n*) is the unit vector in the surface normal direction:

_{s}#### 2) Inverse distance weighted interpolation

*c*, given by Eq. (7), are a function of radial distance from the interpolation point, which in our method is the image point. In Eq. (7),

*R*

_{max}is the maximum radius from the interpolation point of the group of neighbors, and

*R*and

_{n}*c*are the radial distance and weighting coefficient for the

_{n}*n*th neighbor:

Equation (7), which is plotted in Fig. 2, produces an infinite weight for a node that is coincident with the image point, while the node located farthest away at *R*_{max} is used to define the sphere of influence for the interpolation, but has no influence itself with a weighting factor of zero. The variable *p* is a power exponent that controls the rate of decay of the weighting coefficient with increasing radial distance. Gao et al. (2007) used a power parameter of 2. Numerical instabilities occurred for certain geometries of cut cells in our simulations with the power parameter set to 2. These instabilities arose when the ghost cell was far from the boundary (≈80%–90% Δ*x*), so that once reflected across the boundary the image point was much closer to the second fluid node from the boundary than the first. In this case, the weighting coefficient for the second fluid node was very large, and influenced the solution so disproportionately that the value at the ghost point was nearly equal in magnitude to the value of the second fluid node. The first fluid node was essentially decoupled and had no influence on the interpolation. The node’s value could then grow in magnitude and destabilize the entire solution. These instabilities were eliminated completely by using a parameter of either 1 or ½, so that each node in the interpolation contributed more equally to the interpolation. For the simulations presented here, *p* is set to ½.

Weighting coefficients *c _{n}* for inverse distance weighted interpolation as a function of radius, normalized by the maximum radius

*R*

_{max}. Three different power parameters (

*p*= 2, 1, and 0.5) are shown here.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Weighting coefficients *c _{n}* for inverse distance weighted interpolation as a function of radius, normalized by the maximum radius

*R*

_{max}. Three different power parameters (

*p*= 2, 1, and 0.5) are shown here.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Weighting coefficients *c _{n}* for inverse distance weighted interpolation as a function of radius, normalized by the maximum radius

*R*

_{max}. Three different power parameters (

*p*= 2, 1, and 0.5) are shown here.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

In our algorithm, the first step in identifying neighbors to be used in the interpolation is searching the 64 computational nodes surrounding the image point (i.e., a distance of 2 nodes in each direction from the image point, which is not coincident with a computational node, which defines a 4^{3} cloud of nodes surrounding the image point). Potential neighbors are identified as those residing in the fluid domain, eliminating those nodes in the solid domain. Potential neighbors are then sorted by increasing radial distance, and eight neighbors are chosen by proximity for use in the interpolation. The particular choice of eight neighbors is arbitrary, and was chosen here because eight neighbors are used in the trilinear interpolation algorithm. Gao et al. (2007) note that they found that 3 to 4 nodes in two dimensions and 4 to 5 nodes in three dimensions provided sufficient accuracy.

When Dirichlet boundary conditions are used, the first neighbor is on the immersed boundary along the surface normal vector connecting the image and ghost points. The remaining seven neighbors are computational nodes, as illustrated in Fig. 3a. Inverse distance weighting preserves maxima and minima, even during extrapolation. Therefore, it is guaranteed that the interpolated image point value will be bounded by the values of the boundary condition and neighboring computational nodes.

Eight neighbors marked by squares are chosen for use in determining the coefficients of the inverse distance weighting interpolant. Points for (a) a Dirichlet boundary condition and (b) a Neumann boundary condition are shown.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Eight neighbors marked by squares are chosen for use in determining the coefficients of the inverse distance weighting interpolant. Points for (a) a Dirichlet boundary condition and (b) a Neumann boundary condition are shown.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Eight neighbors marked by squares are chosen for use in determining the coefficients of the inverse distance weighting interpolant. Points for (a) a Dirichlet boundary condition and (b) a Neumann boundary condition are shown.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

When Neumann boundary conditions are imposed, a boundary point is not used because the value on the immersed surface is unknown. In this case, all eight neighbors are computational nodes. Without a point on the boundary, the image point can lie outside of the interpolation region, usually when the ghost node is near the surface, as in Fig. 3b. If extrapolation is used, the calculated value of the image point may not properly account for gradients in the variable field because the extrapolated value will be bounded by the values at the neighbors. In this case, the image point is modified by relocating it to be at the intersection of the surface normal with a face of the computational cut cell, as shown in Fig. 3b. Although the image point is no longer a true reflected image, the relationship given in Eq. (3b) can still be used to achieve the Neumann boundary condition, where

In either case (Dirichlet or Neumann), if eight or fewer potential neighbors exist, as may be the case for convex surfaces in an urban environment, then the sorting routine is skipped, and the algorithm proceeds using all available neighbors within the search area that also reside in the fluid domain. As with trilinear interpolation, once the value at the image point is calculated, the last step is to enforce the boundary condition by calculating and assigning the ghost point value.

Our inverse distance weighted IBM differs significantly from the method proposed in Gao et al. (2007). Our method retains the use of an image point, whose value is found through interpolation, and a relationship between the image point and the boundary condition is used to solve for the ghost point value. Gao et al. (2007) use the inverse distance weighting method along with a Taylor series expansion, where the origin of the Taylor series is the interpolation point, and is located on the immersed boundary (not at an image point). For Dirichlet boundary conditions the variable value on the boundary is known, so that inverse distance weighting is used to reconstruct both first and second derivatives on the boundary (origin of the series) for the first- and second-order terms in the Taylor series expansion. Therefore, the inverse distance weighting interpolation scheme must be applied once for each term in the Taylor series, which is 5 times for each ghost point for two-dimensional cases

## 3. Verification case

In this section, we verify the implementation of our three-dimensional immersed boundary method in the WRF model, and evaluate the accuracy of the two interpolation methods. Verification is performed by comparing simulations of flow over a three-dimensional hill with the native terrain-following coordinate and with each of the immersed boundary methods.

### a. Model setup and initialization

*h*is defined by the Witch of Agnesi curve [Eq. (8)], using a peak mountain height

_{t}*h*of 350 m and a mountain half-width

_{p}*a*of 800 m:

^{−1}for

*u*and 0 m s

^{−1}for

*υ*. Flow is driven by a pressure gradient that would balance a geostrophic wind of 10 m s

^{−1}in the

*x*direction. The Coriolis parameter

*f*is set to a constant value of 1 × 10

^{−4}s

^{−1}. The number of grid points in each direction is (

*nx*,

*ny*,

*nz*) = (60, 60, 91) for the terrain-following case, and (

*nx*,

*ny*,

*nz*) = (60, 60, 95) for the cases using the immersed boundary method. Four additional points are used in the vertical direction in the immersed boundary method to account for the fact that nodes are needed underneath the terrain, therefore the domain begins at

*z*= −200 m in the IBM cases (rather than at

*z*= 0 m in the terrain-following case). In the horizontal dimensions, a constant 100-m grid spacing is used. In the vertical dimension, the grid points are equally spaced in the pressure-based coordinate, so that at initialization Δ

*z*

_{min}= 38.8 m and Δ

*z*

_{max}= 61.3 m for the terrain-following coordinate simulation and Δ

*z*

_{min}= 41.3 m and Δ

*z*

_{max}= 62.0 m for the IBM simulations. Differences in the

*z*grid spacing are unavoidable because of the grid transformation in WRF. The top of the domain is located at a height of 4500 m. A constant eddy viscosity of 20 m

^{2}s

^{−1}is used.

Periodic boundary conditions are used at the lateral boundaries. A no-slip boundary condition is set on velocity at the terrain surface, along with a zero flux condition on temperature. At the top of the domain, the native WRF boundary condition is used (isobaric and a material surface), with a Rayleigh damping layer that acts only on vertical velocity within the top 500 m of the domain.

### b. Results

The flow is integrated for 3 days (72 h), and inertial oscillations are present as the pressure gradient, Coriolis force, and surface friction terms come into balance. These oscillations damp with time, as shown in Fig. 4. The oscillations for the terrain-following and IBM cases are nearly equal in magnitude and phase, so much so that it is difficult to distinguish the individual lines. At 72 h the oscillations are significantly damped, and the solution is approaching a steady state. Results presented in the remainder of this section are instantaneous at a time of 72 h.

Domain-averaged values of *u* and *υ* plotted as a function of time for the original WRF coordinate and for IBM-WRF using trilinear and inverse distance weighted interpolations. Points underneath the terrain (for the IBM cases) are excluded from the domain average. Solutions using the two IBMs show excellent agreement with the terrain-following coordinate case, as the thin gray lines representing IBM lie in the middle of the thick black line representing the solution using terrain-following coordinates. This makes the three lines for each velocity component (*u* and *υ*) nearly indistinguishable.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Domain-averaged values of *u* and *υ* plotted as a function of time for the original WRF coordinate and for IBM-WRF using trilinear and inverse distance weighted interpolations. Points underneath the terrain (for the IBM cases) are excluded from the domain average. Solutions using the two IBMs show excellent agreement with the terrain-following coordinate case, as the thin gray lines representing IBM lie in the middle of the thick black line representing the solution using terrain-following coordinates. This makes the three lines for each velocity component (*u* and *υ*) nearly indistinguishable.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Domain-averaged values of *u* and *υ* plotted as a function of time for the original WRF coordinate and for IBM-WRF using trilinear and inverse distance weighted interpolations. Points underneath the terrain (for the IBM cases) are excluded from the domain average. Solutions using the two IBMs show excellent agreement with the terrain-following coordinate case, as the thin gray lines representing IBM lie in the middle of the thick black line representing the solution using terrain-following coordinates. This makes the three lines for each velocity component (*u* and *υ*) nearly indistinguishable.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

For a quantitative comparison, the WRF and IBM solutions are interpolated onto a common time invariant terrain-following grid, so that they may be compared directly. This new grid uses the same horizontal spacing as the grid for the solution; however, the vertical grid spacing is independent of the computational grid. The IBM solution is subtracted from the WRF solution for each variable, and the maximum magnitudes of the differences in velocity are included in Table 1. The minimum differences, which are not presented in the table, are negligible so that the solutions are nearly identical in those locations. Not surprisingly, minimum differences occur in the top portion of the domain, where the terrain has little effect on the solution, indicating that the balance of forces due to the geostrophic wind is the same between the three simulations. The maximum differences are all located at 216 ≤ *z* ≤ 758 m, which is near the terrain peak of *h _{p}* = 350 m. The maximum differences between the WRF and IBM solutions are similar for the trilinear and inverse distance weighed interpolation schemes, but suggest that the trilinear interpolation scheme is slightly more accurate than the IDW scheme.

The locations in the three-dimensional hill domain of the maximum difference between the WRF and IBM solutions for each velocity component. Velocity for each solution is given at that location, along with absolute and relative differences.

Contours of velocity magnitude, along with quivers indicating the velocity direction are shown at a height of 250 m in Fig. 5 for each simulation. Gray markers are placed at the locations in *x* and *y* of the maximum differences from Table 1. Dark gray markers are used for trilinear interpolation and light gray is used for IDW. Both sets of markers, from trilinear and IDW interpolations, are included on the figure for the WRF solution using terrain-following coordinates. Contours and quivers show that the three solutions are similar. The largest differences occur in the lee of the hill when using trilinear interpolation, but just in front of the hill using the inverse distance weighted method.

Contours of velocity (*u*, *υ*, and *w*) magnitude (m s^{−1}) and quivers of velocity (*u* and *υ* only) direction at a height of 250 m and a time of 72 h for (top) WRF, (middle) IBM-WRF trilinear, and (bottom) IBM-WRF IDW. Flow is over a three-dimensional hill. Gray dots indicate the (*x*, *y*) location of the largest differences from Table 1, with dark and light gray representing trilinear and IDW interpolations, respectively. The black lines indicate the locations of the velocity profiles shown in Fig. 6.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Contours of velocity (*u*, *υ*, and *w*) magnitude (m s^{−1}) and quivers of velocity (*u* and *υ* only) direction at a height of 250 m and a time of 72 h for (top) WRF, (middle) IBM-WRF trilinear, and (bottom) IBM-WRF IDW. Flow is over a three-dimensional hill. Gray dots indicate the (*x*, *y*) location of the largest differences from Table 1, with dark and light gray representing trilinear and IDW interpolations, respectively. The black lines indicate the locations of the velocity profiles shown in Fig. 6.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Contours of velocity (*u*, *υ*, and *w*) magnitude (m s^{−1}) and quivers of velocity (*u* and *υ* only) direction at a height of 250 m and a time of 72 h for (top) WRF, (middle) IBM-WRF trilinear, and (bottom) IBM-WRF IDW. Flow is over a three-dimensional hill. Gray dots indicate the (*x*, *y*) location of the largest differences from Table 1, with dark and light gray representing trilinear and IDW interpolations, respectively. The black lines indicate the locations of the velocity profiles shown in Fig. 6.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Black lines appearing in the middle of the domain near the peak of the hill (*y* ≈ 3000 m), and three-fourths though the domain in the lee of the hill (*y* ≈ 4500 m) indicate the location of the velocity profiles shown in Fig. 6. At each *y* location there are five vertical profiles of *u*, *υ*, and *w* plotted at different *x* locations (1000, 2000, 3000, 4000, and 5000 m).The locations of these velocity profiles are representative of the regions that differ most. While the maximum differences in Table 1 may seem large, the velocity profiles in Fig. 6 put these numbers in perspective by illustrating how similar the solutions are. Profiles of each velocity component are shown for several locations along the *x* dimension. Instantaneous velocity profiles are nearly identical for the WRF and two IBM solutions. Differences in the solutions are attributable to two main causes: differences in the vertical grid and slight differences in the phase of the inertial oscillation. Each of these causes is discussed further below.

Instantaneous profiles of (a),(b) *u*, (c),(d) *υ*, and (e),(f) *w* velocity are shown for flow over a three-dimensional hill. Profiles are plotted at several horizontal locations located along a slice in the *y* dimension (slice locations are shown in Fig. 5). Profiles are located at (left) *y* ≈ 3000 m and (right) *y* ≈ 4500 m.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Instantaneous profiles of (a),(b) *u*, (c),(d) *υ*, and (e),(f) *w* velocity are shown for flow over a three-dimensional hill. Profiles are plotted at several horizontal locations located along a slice in the *y* dimension (slice locations are shown in Fig. 5). Profiles are located at (left) *y* ≈ 3000 m and (right) *y* ≈ 4500 m.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Instantaneous profiles of (a),(b) *u*, (c),(d) *υ*, and (e),(f) *w* velocity are shown for flow over a three-dimensional hill. Profiles are plotted at several horizontal locations located along a slice in the *y* dimension (slice locations are shown in Fig. 5). Profiles are located at (left) *y* ≈ 3000 m and (right) *y* ≈ 4500 m.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

The largest differences (based on percentage) from Table 1 are in the *w* velocity component. These differences can also be seen in the profiles of *w* velocity, especially those located at *y* ≈ 4500 m. These small velocities are sensitive to the vertical grid. Additional simulations were run to examine this sensitivity, including terrain-following coordinate simulations with a constant vertical grid spacing of Δ*z* = 50 or 100 m and an IBM-WRF simulation with a constant Δ*z* of 50 m. When terrain-following coordinates are used, the vertical grid spacing is constant over terrain that is 0 m in height, but necessarily compressed over elevated terrain (such as the hill). The IBM-WRF simulation used trilinear interpolation. Results from these simulations are included in Fig. 7 at two different times (72 and 88 h). Results for *u* and *υ* appear as in Fig. 6, and are therefore not shown. Profiles from the WRF simulations (shown as solid lines) show variation when the vertical grid spacing is changed from 50 to 100 m. This variation in Fig. 4 is as large or larger than the differences seen between the WRF and IBM-WRF simulations, making those differences reasonable.

Profiles of *w* are shown for two WRF and IBM-WRF simulations of flow over a three-dimensional hill. Results are shown for different vertical grids (Δ*z* = 50 m and Δ*z* = 100 m) at several horizontal locations located along a slice in the *y* dimension at *y* ≈ 4500 m. Profiles are shown (a) after 72 h of integration (as in Fig. 6) and (b) at 88 h. This illustrates that small velocities, such as the *w* velocity shown here, are sensitive to changes in the vertical grid and unsteadiness due to the inertial oscillations.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Profiles of *w* are shown for two WRF and IBM-WRF simulations of flow over a three-dimensional hill. Results are shown for different vertical grids (Δ*z* = 50 m and Δ*z* = 100 m) at several horizontal locations located along a slice in the *y* dimension at *y* ≈ 4500 m. Profiles are shown (a) after 72 h of integration (as in Fig. 6) and (b) at 88 h. This illustrates that small velocities, such as the *w* velocity shown here, are sensitive to changes in the vertical grid and unsteadiness due to the inertial oscillations.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Profiles of *w* are shown for two WRF and IBM-WRF simulations of flow over a three-dimensional hill. Results are shown for different vertical grids (Δ*z* = 50 m and Δ*z* = 100 m) at several horizontal locations located along a slice in the *y* dimension at *y* ≈ 4500 m. Profiles are shown (a) after 72 h of integration (as in Fig. 6) and (b) at 88 h. This illustrates that small velocities, such as the *w* velocity shown here, are sensitive to changes in the vertical grid and unsteadiness due to the inertial oscillations.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Furthermore, from the perspective given in Fig. 4, the WRF and IBM-WRF simulations are in an identical phase of the inertial oscillation. However, when the velocity profiles are viewed in time, it can be seen that the velocity profiles are sensitive to this unsteadiness. Profiles of *w* velocity are shown at 72 h on the left in Fig. 7 and 88 h on the right. From this figure, it can be seen that the *w* velocity components oscillate around a magnitude of zero, and the differences in the model results depend on this oscillation. For example, at 88 h the IBM-WRF solution matches well with the higher-resolution (Δ*z* = 50 m) WRF simulation, while the coarser WRF simulation appears to be different.

Domain-averaged velocity values are included in Table 2 for each velocity component along with the domain-averaged absolute value of the differences between the WRF and IBM solutions. When the model predictions are viewed in an averaged sense, the differences are slightly larger for IBM-WRF IDW than for IBM-WRF trilinear, but all of the differences are relatively small. While IBM-WRF IDW appears to be the least favorable of the two IBM options for this flow case, the method has additional favorable properties, discussed earlier. For very complex geometries, such as the urban geometry in the following section, the trilinear interpolation algorithm fails. Common problems were the inability to find eight appropriate neighbors and ill-conditioned Vandermonde matrices. Additionally, in some cases (such as at corners of buildings) the direction of the flux boundary condition is ambiguous or prescribed in an unintended direction. For these reasons, the IDW core of the IBM is use in the remainder of this work.

Domain-averaged velocity values and the absolute value of the differences for each velocity component in the simulations of flow over a three-dimensional hill. All quantities have units of m s^{−1}.

## 4. Flow in urban environments

The Joint Urban 2003 (JU2003) field campaign took place in Oklahoma City over a period of 1 month, and is detailed in Allwine and Flaherty (2006). This field campaign provides an excellent test case for verifying the use of the IBM in a numerical weather prediction model. Over 20 institutions participated in the field study, providing an extensive set of urban atmospheric flow and dispersion observations. Simulations of the first continuous release from IOP3 from this field campaign are presented in this section. IOP3 was chosen due to the favorable steady wind conditions and the availability of previously published model results from the urban computational fluid dynamics code, Finite Element Model in 3-Dimensions and Massively Parallelized (FEM3MP), for comparison. IOP3 is a daytime observation period occurring on 7 July 2003. The first continuous release occurred during a 30-min period near the Botanical Gardens from 1600 to 1630 UTC. Our setup utilizes the one-way nesting capabilities in WRF, and additionally uses a large-eddy simulation (LES) turbulence closure. Both of these features are new to our IBM-WRF implementation.

Our results are compared to the observations from the field campaign, and additionally with model predictions from Chan and Leach (2007), where the FEM3MP finite-element model (documented in Gresho and Chan 1998) is used to simulate the same release. The FEM3MP simulation uses a high-resolution (~1 m near solid boundaries) structured body-fitted grid with a Reynolds averaged Navier–Stokes (RANS) turbulence closure.

### a. Model setup and initialization

In this simulation, a domain with the Oklahoma City terrain is nested within a parent domain with flat terrain. Flow is driven in the parent domain with a pressure gradient and periodic lateral boundary conditions are used, so that the flow on the parent domain is fully developed turbulent channel flow over flat terrain. One-way nesting is used in a concurrent simulation run, so that boundary conditions (with the exception of boundary conditions for passive scalars) and resolved turbulent fluctuations are passed from the channel flow domain to the Oklahoma City domain at the frequency of the parent domain time step. This setup is similar to that used in Golaz et al. (2009) where flow over Askervein Hill is simulated using the Coupled Ocean–Atmosphere Mesoscale Prediction System (COAMPS). The setup in Golaz et al. (2009) uses an outer domain with flat terrain, while using an inner domain with the Askervein Hill terrain, however, terrain-following coordinates are used on both domains. Our setup uses the IBM on both domains. The use of nested domains has two main purposes. First, it provides fully developed turbulent inflow conditions to the Oklahoma City domain, and thus eliminates the need for periodic boundary conditions on the domain with urban terrain, which are only appropriate for surface conditions that are homogeneous or periodic (such as the conditions on the outer domain). Second, the use of one-way nesting allows us to examine how IBM works with nested domain configurations, where terrain is not resolved on outer domains but is explicitly resolved on the innermost domain using IBM.

A two-dimensional array of terrain heights for the Oklahoma City case is created by overlaying the horizontal WRF grid for the nested domain with an Environmental Systems Research Institute (ESRI) shapefile of the downtown region. A shapefile spatially describes geometries as sets of points, polylines, or polygons. For the Oklahoma City shapefile, the buildings are defined as sets of overlapping polygons, where each polygon is assigned a unique height. Once the WRF grid is overlaid on the shapefile, the building heights that are coincident with the nodes on the WRF grid are sampled to create a two-dimensional array of terrain heights, shown in Fig. 8. The computational grid is also visualized using two cross sections in Fig. 8. In this case, the terrain is sampled at a horizontal resolution of 1 m, and the computational grid has a horizontal resolution of 2 m.

Three-dimensional terrain for the Oklahoma City domain (the inner WRF nest), where the buildings are explicitly resolved in the WRF simulation using the IBM. Two cross sections of the computational grid are shown. Every tenth grid line is included for visual clarity.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Three-dimensional terrain for the Oklahoma City domain (the inner WRF nest), where the buildings are explicitly resolved in the WRF simulation using the IBM. Two cross sections of the computational grid are shown. Every tenth grid line is included for visual clarity.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Three-dimensional terrain for the Oklahoma City domain (the inner WRF nest), where the buildings are explicitly resolved in the WRF simulation using the IBM. Two cross sections of the computational grid are shown. Every tenth grid line is included for visual clarity.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

The outer domain is initialized and spun up for several hours until fully developed turbulent channel flow is achieved. Once the flow is developed, the inner nest is initialized and also spun up before the continuous release of IOP3 is simulated. The flat terrain on the outer domain, located at a height of 0 m, is represented with the IDW IBM. Terrain-following coordinates can also be used on the parent domain, in lieu of the IBM. The ability to nest an IBM domain within terrain-following coordinates has been implemented, but is not used in this case, which is an IBM domain with complex urban geometry nested within another IBM domain with flat terrain.

For the outer domain, the number of grid points in each direction is (*nx*, *ny*, *nz*) = (97, 124, 170). In the horizontal directions, a constant 6-m grid spacing is used. In the vertical dimension, the grid points are stretched over the domain height of 407 m, which spans *z* = −3 to 407 m. The vertical grid spacing is a constant Δ*z* = 1 m below a height of 10 m, and is Δ*z* = 3 m above 150 m. Hyperbolic tangent stretching is used for Δ*z* between 10 and 150 m. These dimensions result in a total domain size of 576 × 738 × 410 m^{3}. The Runge–Kutta time step is Δ*t* = *w* only is used in the top 50 m of the domain. The standard Smagorinsky turbulence model in the WRF distribution is used, and Coriolis forcing is neglected.

A grid nesting ratio of 1:3 is used, so that the nested Oklahoma City domain has a horizontal resolution of Δ*x* = Δ*y* = 2 m, and a time step of Δ*t* = *nx*, *ny*, *nz*) = (259, 340, 170) grid points, for a total size of 516 × 678 × 410 m^{3}. The nested domain starts on the sixth grid point of the parent domain in both the *x* and *y* directions. The parent simulation is conducted for the sole purpose of providing turbulent channel flow data at the lateral boundaries; therefore, our parent domain is only slightly larger than our nested domain to minimize the required computational cost.

With regards to computational cost, it should be noted that this simulation was computationally intensive. The simulations were completed on 256 AMD Opteron 2.3-GHz processors. On this system one minute of simulation time for both domains takes about 7 h of wall clock time, meaning that the simulation of the 30-min release in IOP3 took 210 h in addition to the time needed for spinup, which was substantial. A major constraint is the domain decomposition method used in WRF, which limits the number of processors that can be used. The parent domain uses the maximum number of processors (16 × 16) without being overdecomposed, meaning at least 6 grid points per processor in each horizontal dimension. The nested domain is decomposed onto the same processors, adding an additional ~15 million nodes to the simulation beyond the ~2 million nodes from the parent domain. Additionally, opportunities exist for optimization of the IBM routines. For example, to maximize accuracy, the IBM interpolants are calculated at each acoustic time step, as detailed in Lundquist et al. (2010a). This process could be streamlined by diminishing the frequency at which the interpolants are calculated, while still maintaining sufficient accuracy.

When the nested (urban) domain is initialized, the solution from the parent (flat) domain is interpolated onto the nested grid. The inverse distance weighted IBM is used to represent the Oklahoma City terrain, and the velocities within the solid domain (within buildings) are set to zero when the nest is initially spawned. Additionally, the IBM routines are used to impose no-slip boundary conditions on the buildings before integration of the nest begins.

For the outer domain, the atmosphere is initialized with a neutral temperature profile, which Lundquist and Chan (2007) show to be a valid assumption for IOP3 as a result of the dominance of mechanical building-induced dispersion in the urban core. Flow is driven with a constant pressure gradient with components in both the *x* and *y* directions to achieve the correct wind speed and direction, and periodic lateral boundary conditions are used. A target wind speed and direction profile for the scalar release site, shown in Fig. 9 as the solid gray line, is developed with time-averaged data from instruments in the immediate vicinity of the Botanical Gardens scalar release location. The pressure gradient for the simulation is chosen so that the average velocity profile and direction at the scalar release site on the inner nest closely resembles the target profile. Instruments used to create the target profile include Dugway Proving Grounds Portable Weather Information Display Systems (DPG PWIDS) numbers 10 and 11 with propeller and vane anemometers, DPG SuperPWIDS numbers 17 and 20 with 3D sonic anemometers, Argonne National Laboratory’s minisodar, and Air Resources Laboratory Field Research Division’s (ARLFRD) 3D anemometer collocated with the release location. The PWIDS and SuperPWIDS are located at an elevation of 8 m, the minisodar data range in elevation from 15 to 135 m, and the release site anemometer is located at a height of 2 m. Data are time averaged over the first continuous release period from 1600 to 1630 UTC.

The time-averaged (left) velocity and (right) wind direction profiles as measured by instrumentation in the immediate vicinity of the IOP3 scalar release (gray solid line) and in the IBM-WRF simulation at the release site (black dashed line). The pressure gradient on the outer domain is adjusted to achieve this agreement at the scalar release site.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

The time-averaged (left) velocity and (right) wind direction profiles as measured by instrumentation in the immediate vicinity of the IOP3 scalar release (gray solid line) and in the IBM-WRF simulation at the release site (black dashed line). The pressure gradient on the outer domain is adjusted to achieve this agreement at the scalar release site.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

The time-averaged (left) velocity and (right) wind direction profiles as measured by instrumentation in the immediate vicinity of the IOP3 scalar release (gray solid line) and in the IBM-WRF simulation at the release site (black dashed line). The pressure gradient on the outer domain is adjusted to achieve this agreement at the scalar release site.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

The time-averaged wind speed and direction profiles from the IBM-WRF simulation are shown in Fig. 9 as dashed black lines. Profiles from the simulation are located at the release site, and are time averaged over the 30-min release interval with model output at a frequency of 20 s. Velocity predictions from the simulation match the observations well near the surface (at *z* < 10 m), and aloft at 100 m, but overpredict the average velocity between 8 and 100 m. The maximum deviation occurs at an elevation of 15 m, where the mean wind speed in the simulation is 3.44 m s^{−1} greater than the observation, while the mean bias over the profile is 0.65 m s^{−1}. This overprediction could be caused by the fact that the fetch upwind of the scalar release site is short in our domain (see Fig. 10), meaning that the effect of upwind obstacles (primarily buildings and vegetation) is not captured.

A snapshot of the *υ* component of velocity (m s^{−1}) for the (left) outer nest and (right) inner nest at a height of ≈9.5 m. The dashed line in the outer domain indicates the horizontal extent of the inner nest. The outer nest simulates a neutral atmospheric boundary layer, while the inner domain simulates flow through downtown Oklahoma City. The scalar release location for JU2003 IOP3 is marked with a star, and the prevailing wind is from the southwest direction.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

A snapshot of the *υ* component of velocity (m s^{−1}) for the (left) outer nest and (right) inner nest at a height of ≈9.5 m. The dashed line in the outer domain indicates the horizontal extent of the inner nest. The outer nest simulates a neutral atmospheric boundary layer, while the inner domain simulates flow through downtown Oklahoma City. The scalar release location for JU2003 IOP3 is marked with a star, and the prevailing wind is from the southwest direction.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

A snapshot of the *υ* component of velocity (m s^{−1}) for the (left) outer nest and (right) inner nest at a height of ≈9.5 m. The dashed line in the outer domain indicates the horizontal extent of the inner nest. The outer nest simulates a neutral atmospheric boundary layer, while the inner domain simulates flow through downtown Oklahoma City. The scalar release location for JU2003 IOP3 is marked with a star, and the prevailing wind is from the southwest direction.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

In the JU2003 field campaign, continuous releases of sulfur hexafluoride (SF_{6}) were made. Releases during IOP3 were located at 634603E, 3925763N in universal transverse Mercator (UTM) coordinates (m) at a height of 2 m AGL. The scalar release is simulated using IBM-WRF once the flow is spun up in both domains. The ability to simulate a continuous source was added to WRF as a part of this work. Zero gradient boundary conditions are used on the passive scalar at the immersed boundaries. Additionally, an option for flow-dependent boundary conditions for the passive scalar was added for nested domains at the lateral boundaries, even when specified (nested) boundary conditions are used for the other prognostic variables. With flow-dependent boundary conditions, the scalar cloud is allowed to exit the nested domain, and no passive scalars exist on the parent domain.

### b. Results for flow through urban terrain

Figure 10 shows instantaneous contours of *υ* velocity from a top view for the parent and nested domains at the beginning of the scalar release period. This view is at a height of approximately 9.5 m, and the extent of the inner nest is depicted by black dashed lines on the outer domain. On the inner nest, the solution is masked by buildings with heights greater than the height of the plane shown. In this figure, the inlet flow is considered to primarily be the southern edge of the domain, as the wind direction is approximately 193°. Here, it can be seen that the velocity features from the outer nest also appear at the inlet of the inner nest. At the outlet (the northern domain edge), the solution on the inner domain is forced to match the solution on the parent domain, even though there is clearly a wake region behind the northern most buildings that is being truncated. This mismatch of velocities and pressure gradient contaminates a portion of the solution near the outlet of the Oklahoma City domain. This is acceptable for our purposes because the scalar release in IOP3 is located in the southern portion of the domain, at (*x*, *y*) = (148 m, 247 m) on the inner nest. This mismatch of the solutions at the outlet of nested simulations should be decreased when using two-way nesting, or when the urban terrain is parameterized on the outer domain. Both modifications to the setup should be explored in the future when computational resources allow.

Figure 11 shows contours of velocity magnitude and quivers indicating flow direction at approximately the height of the scalar release at the beginning of the continuous release period. Many flow features are present in the large-eddy simulation, including high speed jets at contractions of urban canyons and separation zones behind buildings. A similar figure depicting the averaged flow field appears as Fig. 5 in Chan and Leach (2007) for their RANS; however, the figure for our LES shows instantaneous results and therefore illustrates the flow variability due to the resolved turbulence.

Contours of velocity magnitude (m s^{−1}) and quivers indicate flow direction for an instantaneous time at the beginning of the Oklahoma City JU2003 IOP3 scalar release. This figure illustrates the complexities of the flow modeled by the large-eddy simulation. This top view for a portion of the domain is at a height of *z* ≈ 2.5 m, and a star marks the location of the scalar release. Every third wind vector is shown.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Contours of velocity magnitude (m s^{−1}) and quivers indicate flow direction for an instantaneous time at the beginning of the Oklahoma City JU2003 IOP3 scalar release. This figure illustrates the complexities of the flow modeled by the large-eddy simulation. This top view for a portion of the domain is at a height of *z* ≈ 2.5 m, and a star marks the location of the scalar release. Every third wind vector is shown.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Contours of velocity magnitude (m s^{−1}) and quivers indicate flow direction for an instantaneous time at the beginning of the Oklahoma City JU2003 IOP3 scalar release. This figure illustrates the complexities of the flow modeled by the large-eddy simulation. This top view for a portion of the domain is at a height of *z* ≈ 2.5 m, and a star marks the location of the scalar release. Every third wind vector is shown.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Figure 12 includes time-averaged wind speed vectors from both the simulation and the field campaign over the 30-min release interval. Data from the simulation are stored at a frequency of 20 s, so that 90 instances contribute to the time-averaged wind speed. Field campaign instruments used for the comparison are the DPG PWIDS numbers 4, 5, 7, 8, 11, and 12, as well as SuperPWIDS numbers 7 through 20 (abbreviated as P and SP in the following discussion). Five of these sites have collocated instruments. The PWIDS normally collected and averaged data at 10-s intervals; however, during IOP3, 1-min averages were recorded due to data collection problems. Thirty 1-min averages are used to calculate the 30-min-average observations in Fig. 12. SuperPWIDS sonic anemometer observations are collected at 10 Hz, and time averaged over the 30-min interval. Wind magnitude and direction are recorded for the PWIDS; however, velocity components (*u*, *υ*, *w*) are recorded for the SuperPWIDS and the simulation. The time-averaging procedure used here calculates the wind speed and direction at each time from instantaneous velocity components, and then takes a scalar average of wind speed and direction.

30-min time-averaged velocity vectors are shown for the simulation (with arrow heads) and for observations from JU2003 IOP3 (without arrow heads) at a height of 8 m. DPG PWIDS using 2D cup anemometers are shown in black, while SuperPWIDS using 3D sonic anemometers are shown in gray. The instruments are labeled with the instrument number (P = PWID, SP = SuperPWID).

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

30-min time-averaged velocity vectors are shown for the simulation (with arrow heads) and for observations from JU2003 IOP3 (without arrow heads) at a height of 8 m. DPG PWIDS using 2D cup anemometers are shown in black, while SuperPWIDS using 3D sonic anemometers are shown in gray. The instruments are labeled with the instrument number (P = PWID, SP = SuperPWID).

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

30-min time-averaged velocity vectors are shown for the simulation (with arrow heads) and for observations from JU2003 IOP3 (without arrow heads) at a height of 8 m. DPG PWIDS using 2D cup anemometers are shown in black, while SuperPWIDS using 3D sonic anemometers are shown in gray. The instruments are labeled with the instrument number (P = PWID, SP = SuperPWID).

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Observed and predicted wind speed are given in Table 3 for the plane of 8 m AGL. P11 and SP17 are collocated, and are the instruments nearest to the release location (with the exception of the ARLFRD 3D sonic anemometer located at the release site at 2 m AGL). The simulated wind speed at this location is larger than the observed wind speed (44% for P11 and 12% for SP17), while the wind direction is consistent with observations (within 1° for P11 and 7° for SP17). This is in agreement with the wind speed and direction shown in Fig. 9 for the release location. While there can be significant differences between the model predictions and observations, the observations from collocated instruments also exhibit differences. For instance, the P05, SP14 collocated instruments report a 0.65 m s^{−1} difference in time-averaged wind speed, and the P12, SP18 collocated instruments report a 81° difference in time-averaged wind direction. Still, the differences between the predicted and observed values can be large; for example, there is a 4.98 m s^{−1} wind speed difference at SP07. The SP07 site is located at the intersection of two street canyons, which is one possible reason for the large difference. Additional possible causes for these differences are presented at the end of this section.

Predicted and observed time-averaged wind speed (m s^{−1}) and direction (°) during JU2003 IOP3. Collocated instruments are listed in the same row. All instruments are located at 8 m AGL.

*C*is the predicted time-averaged value,

_{p}*C*is the observed time-averaged value, and the overbar denotes averaging over the dataset:

_{o}Statistical performance metrics for wind speed from the IBM-WRF and FEM3MP (Chan and Leach 2007) simulations of JU2003 IOP3.

*U*| is the predicted wind speed,

_{p}*φ*and

_{p}*φ*are the predicted and observed wind angles, and

_{o}*N*is the number of samples being averaged:

Separate calculations are made for assessing the simulation performance relative to the PWIDS and SuperPWIDS. This avoids more heavily weighting the influence of collocated instrument locations on the metrics. Therefore only 6 instruments contribute to the PWID dataset, while 14 contribute to the SuperPWID dataset. It can be seen in Table 4 that the IBM-WRF simulation compares more closely to the observations from the PWIDS than the SuperPWIDS, as four of the metrics (FAC2, FB, MG, NMSE) for the PWIDS are nearer to the perfect-model values than the SuperPWIDS results. Possible reasons for this could include the location of the instruments or differences in the instrument type (cup vs sonic anemometer). All of the performance metrics, with the exception of the mean geometric bias for the SuperPWIDS, fall within the acceptable range noted in Chang and Hanna (2004). Furthermore, IBM-WRF performs approximately the same as FEM3MP, which is purpose-built for urban modeling.

Figure 13 shows time-averaged contours of scalar concentration from the IBM-WRF simulation at a height of 1 m, along with markers showing the average concentration measured from the LLNL “blue box” instruments. As with the velocity data, the scalar data from the simulation are available at a frequency of 20 s and are averaged over the 30-min interval of the continuous release. The blue box samplers contained a series of 200-cc bags, and drew 10 evenly spaced sips of 20 cc. Two bags were used to sample the air during the first continuous release of IOP3, running from 1600:14 to 1613:17 UTC and 1615:14 to 1628:19 UTC. The concentration values from these two bags are averaged to arrive at the 30-min-averaged observation values. Observations are available at 19 locations during IOP3, 14 of which are located at an elevation of 1 m while the remaining 5 instruments are located at other various elevations. The 14 instruments located at an elevation of 1 m are included in Fig. 13, while the concentration values from the simulation and the observations are given for all of the instruments in Table 5. Additionally, all of the instruments are used to calculate the statistical performance measurements included in Table 6. While the simulation correctly predicts the split plume in front of the nearest building, it overpredicts the concentration at a majority of the instrumented locations (14 out of 19). Possible causes of this over prediction are discussed below, and warrant further investigation.

30-min time-averaged contours of scalar concentration (ppb) are shown for the simulation and data from the LLNL blue boxes are shown as markers for JU2003 IOP3. All observations are at a height of 1 m. Instruments located at other elevations are not shown although they are included in the performance metrics.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

30-min time-averaged contours of scalar concentration (ppb) are shown for the simulation and data from the LLNL blue boxes are shown as markers for JU2003 IOP3. All observations are at a height of 1 m. Instruments located at other elevations are not shown although they are included in the performance metrics.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

30-min time-averaged contours of scalar concentration (ppb) are shown for the simulation and data from the LLNL blue boxes are shown as markers for JU2003 IOP3. All observations are at a height of 1 m. Instruments located at other elevations are not shown although they are included in the performance metrics.

Citation: Monthly Weather Review 140, 12; 10.1175/MWR-D-11-00311.1

Predicted and observed time-averaged concentration (ppb) for JU2003 IOP3. Observations are from the LLNL blue boxes. Easting and northing are in UTM coordinates (m).

Statistical performance metrics for the scalar release from the IBM-WRF and FEM3MP (Chan and Leach 2007) simulations of JU2003 IOP3. Observations are from the LLNL blue boxes.

Statistical performance metrics for scalar concentration values in the IBM-WRF and FEM3MP simulations are given in Table 6. Neither the IBM-WRF nor the FEM3MP simulations meet the criteria defined in Chang and Hanna (2004) for good model performance. IBM-WRF has a larger percentage of points that meet the FAC5 criteria (53% vs 42% in FEM3MP). Chang and Hanna (2004) note that for most atmospheric concentration measurements the distribution is lognormal, and in this case the linear measures of FB and NMSE may be overly influenced by the highest concentrations, and the logarithmic value of MG may provide a more balanced treatment. The values of FB and NMSE are far from a perfect model for IBM-WRF, while the results are somewhat better for MG. Chang and Hanna (2004) note that the FAC2 or FAC5 value is the most robust measurement of performance because it is not overly influenced by outliers in the dataset. IBM-WRF achieves a FAC5 of 0.53; however, the FAC2 value is only 0.16. While there is clearly room for improvement in the results for the wind and scalar fields, the results are consistent with the performance of other urban models (Pullen et al. 2005; Chan and Leach 2007; Burrows et al. 2007; Warner et al. 2008; Gowardhan et al. 2011; Hanna et al. 2011).

Several possibilities exist for the differences seen in both velocities and scalar concentrations between the simulation and observations. It is probable that the default input parameters in WRF could be tuned to increase accuracy. The default Smagorinsky coefficient in WRF (0.25) is larger than the normally recommended value of 0.18. It is also possible that the default Prandtl number (⅓) in WRF is inaccurate. Both of these choices of parameters should be further examined, but exploration of the effects is not possible at this time because of the large computational demands of this simulation. It is also possible that differences between the predicted and observed values can be attributed to simplifications in the simulation. For instance, many of the instruments are located in a complex urban intersections with physical elements (vehicles, vegetation, signage, etc.) that are not captured in the model. Additionally, Klein and Young (2011) analyzed concentration measurements from multiple continuous releases during IOP3, and showed that while the meteorological conditions were relatively constant there was high variability in the concentration statistics, highlighting the difficulty of obtaining statistically repeatable observations.

The IBM-WRF simulation presented here is semi-idealized to facilitate comparisons with a traditional urban CFD model, but use of additional WRF features could improve accuracy. Regional weather effects can be included at the lateral boundaries through the use of grid nesting, and the effects of atmospheric processes can be included by using atmospheric parameterizations. Additionally, the use of more sophisticated LES turbulence closures and two-way nesting can be included in future IBM-WRF simulations.

## 5. Conclusions

We have developed an IBM for the WRF model, which is capable of handling highly complex urban terrain, as demonstrated by our Oklahoma City test case from the JU2003 field experiment. A two-dimensional IB method was previously presented in Lundquist et al. (2010a). We first extended this method into three dimensions, and validated the implementation by simulating flow over a hill and comparing the solution to results achieved using the native terrain-following coordinate. Additionally, a new IB method was developed using an inverse distance weighted interpolation method. This second method was also validated with the canonical case of flow over a three-dimensional hill. We found that while the trilinear interpolation algorithm provided accurate results for flow over a smooth hill and other simple geometries, the algorithm was not robust enough to be used with the real urban terrain from Oklahoma City on a grid with 2-m horizontal resolution. The alternative method based on inverse distance weighted interpolation provided additional flexibility, and was also shown to produce accurate results for the hill test case. Additionally, the method proved to be robust enough to allow simulations of flow over real urban terrain.

IOP3 from the JU2003 field campaign was simulated using IBM-WRF with IDW, and the results were compared to observations as well as to a simulation from the FEM3MP CFD model. The statistical performance metrics achieved for velocity and scalar concentration were close to those documented in Chan and Leach (2007) using body-fitted coordinates. When velocity data were analyzed, IBM-WRF performed well according to criteria established in Chang and Hanna (2004). Results were less accurate for scalar concentration, with only 53% of the predictions being within a factor of 5 (FAC5) of the observations, although the performance metrics were consistent with other urban models.

The IBM-WRF simulation presented here is semi-idealized, which facilitates comparisons with a traditional urban CFD model such as FEM3MP. For example, the inlet flow to the urban domain is that of a fully turbulent neutral boundary layer, rather than nesting the urban domain into a mesoscale simulation, which would provide realistic time-varying boundary conditions. Furthermore, atmospheric stability effects are neglected in this simulation, although they can be treated with IBM-WRF. For example, surface heat fluxes could be included in IBM-WRF by coupling to a three-dimensional radiation model. Regional weather effects, atmospheric stability, the use of sophisticated LES turbulence closures, and two-way nesting can be included in future IBM-WRF simulations and may increase accuracy. The simulation of flow over the Oklahoma City terrain in a one-way nested configuration demonstrates the ability to seamlessly integrate the IBM into the current WRF framework. Work is in progress to add a surface momentum flux parameterization at the IB, enabling simulation of a wide variety of cases with steep terrain ranging from the mesoscale to the microscale.

## Acknowledgments

Thanks are extended to Jeff Mirocha and Branko Kosović for useful input regarding this work. The support of the National Science Foundation Grant ATM-0645784 [FKC] (Physical Meteorology Program) is also gratefully acknowledged. This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.

## REFERENCES

Adcroft, A., C. Hill, and J. Marshall, 1997: Representation of topography by shaved cells in a height coordinate ocean model.

,*Mon. Wea. Rev.***125**, 2293–2315.Allwine, K., and J. Flaherty, 2006: Joint urban 2003: Study overview and instrument locations. Tech. Rep. PNNL-15967, Pacific Northwest National Laboratory, Richland, WA, 92 pp.

Balaras, E., 2004: Modeling complex boundaries using an external force field on fixed Cartesian grid in large-eddy simulations.

,*Comput. Fluids***33**, 375–404.Burrows, D., E. Hendricks, S. Diehl, and R. Keith, 2007: Modeling turbulent flow in an urban central business district.

,*J. Appl. Meteor. Climatol.***46**, 2147–2164.Calhoun, R., F. Gouveia, J. Shinn, S. Chan, D. Stevens, R. Lee, and J. Leone, 2004: Flow around a complex building: Comparisons between experiments and a Reynolds-averaged Navier–Stokes approach.

,*J. Appl. Meteor.***43**, 696–710.Chan, S., and M. Leach, 2007: A validation of FEM3MP with Joint Urban 2003 data.

,*J. Appl. Meteor. Climatol.***46**, 2127–2146.Chang, J., and S. Hanna, 2004: Air quality model performance evaluation.

,*Meteor. Atmos. Phys.***87**, 167–196.Franke, R., 1982: Scattered data interpolation: Tests of some methods.

,*Math. Comput.***38**, 181–200.Gao, T., Y. Tseng, and X. Lu, 2007: An improved hybrid Cartesian/immersed boundary method for fluid-solid flows.

,*Int. J. Numer. Methods Fluids***55**, 1189–1211.Ghias, R., R. Mittal, and H. Dong, 2007: A sharp interface immersed boundary method for compressible viscous flows.

,*J. Comput. Phys.***225**, 528–553.Golaz, J.-C., J. D. Doyle, and S. Wang, 2009: One-way nested large-eddy simulation over the Askervein Hill.

,*J. Adv. Model. Earth Syst.***1**(6), 1–6.Gowardhan, A., E. Pardyjak, I. Senocak, and M. Brown, 2011: A CFD-based wind solver for an urban fast response transport and dispersion model.

,*Environ. Fluid Mech.***11**, 439–464.Gresho, P., and S. Chan, 1998: Projection 2 goes turbulent and fully implicit.

,*J. Comput. Fluid Dyn.***9**, 249–272.Hanna, S., O. Hansen, and S. Dharmavaram, 2004: FLACS CFD air quality model performance evaluation with Kit Fox, MUST, Prairie Grass, and EMU observations.

,*Atmos. Environ.***38**, 4675–4687.Hanna, S., and Coauthors, 2011: Comparisons of JU2003 observations with four diagnostic urban wind flow and Lagrangian particle dispersion models.

,*Atmos. Environ.***45**, 4073–4081.Iaccarino, G., and R. Verzicco, 2003: Immersed boundary technique for turbulent flow simulations.

,*Appl. Mech. Rev.***56**, 331–347.Janjić, Z., 1977: Pressure gradient force and advection scheme used for forecasting with steep and small scale topography.

,*Beitr. Phys. Atmos.***50**, 186–189.Klein, P., and D. Young, 2011: Concentration fluctuations in a downtown urban area. Part I: Analysis of Joint Urban 2003 full-scale fast-response measurements.

,*Environ. Fluid Mech.***11**, 23–42.Klemp, J., W. Skamarock, and O. Fuhrer, 2003: Numerical consistency of metric terms in terrain-following coordinates.

,*Mon. Wea. Rev.***131**, 1229–1239.Lock, S., H. Bitzer, A. Coals, A. Gadian, and S. Mobbs, 2012: Demonstration of a cut-cell representation of 3D orography for studies of atmospheric flows over very steep hills.

,*Mon. Wea. Rev.***140**, 411–424.Lundquist, J., and S. Chan, 2007: Consequences of urban stability for computational fluid dynamics simulations of urban dispersion.

,*J. Appl. Meteor. Climatol.***46**, 1080–1097.Lundquist, K., 2010: Immersed boundary methods for high-resolution simulation of atmospheric boundary-layer flow over complex terrain. Ph.D. thesis, University of California, Berkeley, Berkeley, CA, 159 pp.

Lundquist, K., F. Chow, and J. Lundquist, 2010a: An immersed boundary method for the Weather Research and Forecasting model.

,*Mon. Wea. Rev.***138**, 796–817.Lundquist, K., F. Chow, and J. Lundquist, 2010b: Numerical errors in flow over steep topography: Analysis and alternatives. Preprints,

*14th Conf. on Mountain Meteorology,*Lake Tahoe, CA, Amer. Meteor. Soc., 10.1. [Available online at https://ams.confex.com/ams/14MountMet/techprogram/paper_173801.htm.]Mahrer, Y., 1984: An improved numerical approximation of the horizontal gradients in a terrain-following coordinate system.

,*Mon. Wea. Rev.***112**, 918–922.Mittal, R., and G. Iaccarino, 2005: Immersed boundary methods.

,*Annu. Rev. Fluid Mech.***37**, 239–261.Mittal, R., H. Dong, M. Bozkurttas, F. Najjar, A. Vargas, and A. von Loebbecke, 2008: A versatile sharp interface immersed boundary method for incompressible flows with complex boundaries.

,*J. Comput. Phys.***227**, 4825–4852.Mohd-Yusof, J., 1997: Combined immersed boundary/b-spline methods for simulations of flow in complex geometry. Tech. Rep., Annual Research Briefs, Center for Turbulence Research, NASA Ames/Stanford University, Palo Alto, CA, 317–327.

Peller, N., A. L. Duc, F. Tremblay, and M. Manhart, 2006: High-order stable interpolations for immersed boundary methods.

,*Int. J. Numer. Methods Fluids***52**, 1175–1193.Pullen, J., J. Boris, T. Young, G. Patnaik, and J. Iselin, 2005: A comparison of contaminant plume statistics from a Gaussian puff and urban CFD model for two large cities.

,*Atmos. Environ.***39**, 1049–1068.Schär, C., D. Leuenberger, O. Fuhrer, D. Lüthi, and C. Girard, 2002: A new terrain-following vertical coordinate formulation for atmospheric prediction models.

,*Mon. Wea. Rev.***130**, 2459–2480.Trini-Castelli, S., and T. Reisin, 2010: Evaluation of the atmospheric RAMS model in an obstacle resolving configuration.

,*Environ. Fluid Mech.***10**, 555–576, doi:10.1007/s10652-010-9167-y.Tseng, Y., and J. Ferziger, 2003: A ghost-cell immersed boundary method for flow in complex geometry.

,*J. Comput. Phys.***192**, 593–623.Walko, R., and R. Avissar, 2008: The Ocean–Land–Atmosphere Model (OLAM). Part II: Formulation and tests of the nonhydrostatic dynamic core.

,*Mon. Wea. Rev.***136**, 4045–4062.Warner, S., N. Platt, J. Urban, and J. Heagy, 2008: Comparisons of transport and dispersion model predictions of the Joint Urban 2003 field experiment.

,*J. Appl. Meteor. Climatol.***47**, 1910–1928.Zängl, G., 2002: An improved method for computing horizontal diffusion in a sigma-coordinate model and its application to simulations over mountainous topography.

,*Mon. Wea. Rev.***130**, 1423–1432.Zängl, G., 2003: A generalized sigma-coordinate system for the MM5.

,*Mon. Wea. Rev.***131**, 2875–2884.Zängl, G., L. Gantner, G. Hartjenstein, and H. Noppel, 2004: Numerical errors above steep topography: A model intercomparison.

,*Meteor. Z.***13**, 69–76.