• Andren, A., , A. R. Brown, , P. J. Mason, , J. Graf, , U. Schumann, , C.-H. Moeng, , and F. T. M. Nieuwstadt, 1994: Large-eddy simulation of a neutrally stratified boundary layer: A comparison of four computer codes. Quart. J. Roy. Meteor. Soc., 120, 14571484.

    • Search Google Scholar
    • Export Citation
  • Aoyama, Y., , and J. Nakano, 1999: RS/6000 SP: Practical MPI programming. Tech. Rep. IBM Redbook SG24-5380-00, International Business Machines, 221 pp.

    • Search Google Scholar
    • Export Citation
  • Beare, R. J., and Coauthors, 2006: An intercomparison of large-eddy simulations of the stable boundary layer. Bound.-Layer Meteor., 118, 242272.

    • Search Google Scholar
    • Export Citation
  • Beets, C., , and B. Koren, 1996: Large-eddy simulation with accurate implicit subgrid-scale diffusion. Department of Numerical Mathematics Rep. NM-R9601, Utrecht University, 24 pp.

    • Search Google Scholar
    • Export Citation
  • Betts, A. K., 1974: Reply to comment on the paper ‘Non-precipitating cumulus convection and its parameterization.’ Quart. J. Roy. Meteor. Soc., 100, 469471.

    • Search Google Scholar
    • Export Citation
  • Brasseur, J. G., , and T. Wei, 2010: Designing large eddy simulation of the turbulent boundary layer to capture law-of-the-wall scaling. Phys. Fluids,22, 021303, doi:10.1063/1.3319073.

    • Search Google Scholar
    • Export Citation
  • Bretherton, C. S., and Coauthors, 1999: An intercomparison of radiatively driven entrainment and turbulence in a smoke cloud, as simulated by different numerical models. Quart. J. Roy. Meteor. Soc., 554, 391423.

    • Search Google Scholar
    • Export Citation
  • Bryan, G. H., , J. C. Wyngaard, , and J. M. Fritsch, 2003: Resolution requirements for the simulation of deep moist convection. Mon. Wea. Rev., 131, 23942416.

    • Search Google Scholar
    • Export Citation
  • Celik, I., , M. Klein, , M. Freitag, , and J. Janicka, 2006: Assessment measures for URANS/DES/LES: An overview with applications. J. Turbul., 7, 127.

    • Search Google Scholar
    • Export Citation
  • Chow, F. K., , and P. Moin, 2003: A further study of numerical errors in large-eddy simulations. J. Comput. Phys., 184, 366380.

  • Davis, K. J., , N. Gamage, , C. R. Hagelberg, , D. H. L. C. Kiemle, , and P. P. Sullivan, 2000: An objective method for deriving atmospheric structure from airborne lidar observations. J. Atmos. Oceanic Technol., 17, 14551468.

    • Search Google Scholar
    • Export Citation
  • Deardorff, J. W., 1972a: Numerical investigation of neutral and unstable planetary boundary layers. J. Atmos. Sci., 29, 91115.

  • Deardorff, J. W., 1972b: Three-dimensional numerical modeling of the planetary boundary layer. Workshop on Micrometeorology, D. A. Haugen, Ed., Amer. Meteor. Soc., 271–311.

    • Search Google Scholar
    • Export Citation
  • Deardorff, J. W., 1980: Stratocumulus-capped mixed layers derived from a three-dimensional model. Bound.-Layer Meteor., 18, 495527.

  • de Roode, S. R., , P. G. Duynkerke, , and H. J. J. Jonker, 2004: Large eddy simulation: How large is large enough? J. Atmos. Sci., 61, 403421.

    • Search Google Scholar
    • Export Citation
  • Fedorovich, E., , F. T. M. Nieuwstadt, , and R. Kaiser, 2001: Numerical and laboratory study of a horizontally evolving convective boundary layer. Part I: Transition regimes and development of the mixed layer. J. Atmos. Sci., 58, 7086.

    • Search Google Scholar
    • Export Citation
  • Fedorovich, E., and Coauthors, 2004: Entrainment into sheared convective boundary layers as predicted by different large eddy simulation codes. Preprints, 16th Symp. on Boundary Layer and Turbulence, Portland, ME, Amer. Meteor. Soc., P4.7.

    • Search Google Scholar
    • Export Citation
  • Geurts, B. J., 2001: Modern Simulation Strategies for Turbulent Flow. R. T. Edwards, 327 pp.

  • Geurts, B. J., , and J. Fröhlich, 2002: A framework for predicting accuracy limitations in large-eddy simulation. Phys. Fluids, 14, L41L44.

    • Search Google Scholar
    • Export Citation
  • Gibbs, W. R., 2004: A parallel/recursive algorithm. J. Comput. Phys., 201, 573585.

  • Hatlee, S. C., , and J. C. Wyngaard, 2007: Improved subfilter-scale models from the HATS field data. J. Atmos. Sci., 64, 16941705.

  • Hunt, J. C. R., , J. C. Kaimal, , and J. E. Gaynor, 1988: Eddy structure in the convective boundary layer—New measurements and new concepts. Quart. J. Roy. Meteor. Soc., 482, 827858.

    • Search Google Scholar
    • Export Citation
  • Jonker, H. J. J., , P. G. Duynkerke, , and J. W. M. Cuijpers, 1999: Mesoscale fluctuations in scalars generated by boundary layer convection. J. Atmos. Sci., 56, 801808.

    • Search Google Scholar
    • Export Citation
  • Kaltenbach, H.-J., 1997: Cell aspect ratio dependence of anisotropy measures for resolved and subgrid scale stresses. J. Comput. Phys., 136, 399410.

    • Search Google Scholar
    • Export Citation
  • Kanak, K. M., 2005: Numerical simulation of dust devil–scale vortices. Quart. J. Roy. Meteor. Soc., 131, 12711292.

  • Klemp, J., , and D. Durran, 1983: An upper boundary condition permitting internal gravity wave radiation in numerical mesoscale models. Mon. Wea. Rev., 111, 430444.

    • Search Google Scholar
    • Export Citation
  • Koren, B., 1993: A robust upwind discretization method for advection, diffusion and source terms. Notes on Numerical Fluid Mechanics, Vol. 45, C. B. Vreugdenhil and B. Koren, Eds., Vieweg-Braunschweig, 117–138.

    • Search Google Scholar
    • Export Citation
  • Lele, S. K., 1992: Compact finite difference schemes with spectral-like resolution. J. Comput. Phys., 103, 1642.

  • Lenschow, D. H., , J. C. Wyngaard, , and W. T. Pennell, 1980: Mean-field and second-moment budgets in a baroclinic, convective boundary layer. J. Atmos. Sci., 37, 13131326.

    • Search Google Scholar
    • Export Citation
  • Lenschow, D. H., , J. Mann, , and L. Kristensen, 1994: How long is long enough when measuring fluxes and other turbulence statistics? J. Atmos. Oceanic Technol., 11, 661673.

    • Search Google Scholar
    • Export Citation
  • Lenschow, D. H., , M. Lothon, , S. D. Mayor, , P. P. Sullivan, , and G. Canut, 2011: A comparison of higher-order vertical velocity moments in the convective boundary layer from lidar with in situ measurements and LES. Bound.-Layer Meteor., doi:10.1007/s10546-011-9615-3, in press.

    • Search Google Scholar
    • Export Citation
  • Lilly, D. K., 1967: The representation of small-scale turbulence in numerical simulation experiments. Proc. IBM Scientific Computing Symp. on Environmental Sciences, Yorktown Heights, NY, International Business Machines, 195–210.

    • Search Google Scholar
    • Export Citation
  • Lothon, M., , D. H. Lenschow, , and S. D. Mayor, 2009: Doppler lidar measurements of vertical velocity spectra in the convective planetary boundary layer. Bound.-Layer Meteor., 132, 205226.

    • Search Google Scholar
    • Export Citation
  • Lothon, M., , D. H. Lenschow, , G. Canut, , S. D. Mayor, , and P. P. Sullivan, 2010: Measurements of higher-order turbulence statistics in the daytime convective boundary layer derived from a ground-based Doppler lidar. Proc. Int. Symp. for the Advancement of Boundary Layer Remote Sensing, Paris, France, ISARS. [Available online at http://www.isars2010.uvsq.fr/images/stories/PosterExtAbstracts/P_TUR03_Lothon.pdf.]

    • Search Google Scholar
    • Export Citation
  • Lundquist, K. A., , F. K. Chow, , and J. K. Lundquist, 2010: An immersed boundary method for the Weather Research and Forecasting model. Mon. Wea. Rev., 138, 796817.

    • Search Google Scholar
    • Export Citation
  • Mason, P. J., , and A. R. Brown, 1999: On subgrid models and filter operations in large-eddy simulations. J. Atmos. Sci., 56, 21012114.

    • Search Google Scholar
    • Export Citation
  • Meneveau, C., , and J. Katz, 2000: Scale-invariance and turbulence models for large-eddy simulations. Annu. Rev. Fluid Mech., 32, 132.

  • Meyers, J., , B. J. Geurts, , and P. Sagaut, 2007: A computational error-assessment of central finite-volume discretizations in large-eddy simulation using a Smagorinsky model. J. Comput. Phys., 227, 156173.

    • Search Google Scholar
    • Export Citation
  • Michalakes, J., , J. Dudhia, , D. Gill, , T. Henderson, , J. Klemp, , W. Skamarock, , and W. Wang, 2005: The Weather Research and Forecast Model: Software architecture and performance. Proceedings of the Eleventh ECMWF Workshop on the Use of High Performance Computing in Meteorology, W. Zwieflhofer and G. Mozdzynski, Eds., World Scientific, 156–168.

    • Search Google Scholar
    • Export Citation
  • Mironov, D. V., 2009: Turbulence in the lower troposphere: Second-order closure and mass-flux modelling frameworks. Interdisciplinary Aspects of Turbulence, W. Hillebrandt and F. Kupka, Eds., Lecture Notes in Physics, Vol. 756, Springer-Verlag, 161–221.

    • Search Google Scholar
    • Export Citation
  • Mironov, D. V., , V. M. Gryanik, , C.-H. Moeng, , D. J. Olbers, , and T. H. Warncke, 2000: Vertical turbulence structure and second-moment budgets in convection with rotation: A large-eddy simulation study. Quart. J. Roy. Meteor. Soc., 126, 477515.

    • Search Google Scholar
    • Export Citation
  • Moeng, C.-H., 1984: A large-eddy-simulation model for the study of planetary boundary-layer turbulence. J. Atmos. Sci., 41, 20522062.

    • Search Google Scholar
    • Export Citation
  • Moeng, C.-H., , and J. C. Wyngaard, 1988: Spectral analysis of large-eddy simulations of the convective boundary layer. J. Atmos. Sci., 45, 35733587.

    • Search Google Scholar
    • Export Citation
  • Moeng, C.-H., , and R. Rotunno, 1990: Vertical velocity skewness in the buoyancy-driven boundary layer. J. Atmos. Sci., 47, 11491162.

  • Moeng, C.-H., , and P. P. Sullivan, 1994: A comparison of shear- and buoyancy-driven planetary boundary layer flows. J. Atmos. Sci., 51, 9991022.

    • Search Google Scholar
    • Export Citation
  • Moeng, C.-H., , and P. P. Sullivan, 2002: Large-eddy simulation. Encyclopedia of Atmospheric Sciences, J. R. Holton, J. Pyle, and J. A. Curry, Eds., Academic Press, 1140–1150.

    • Search Google Scholar
    • Export Citation
  • Moeng, C.-H., , M. A. LeMone, , M. F. Khairoutdinov, , S. K. Krueger, , P. A. Bogenschutz, , and D. A. Randall, 2009: The tropical marine boundary layer under a deep convection system: A large-eddy simulation study. J. Adv. Model. Earth Syst., 1 (16), doi:10.3894/JAMES.2009.1.16.

    • Search Google Scholar
    • Export Citation
  • Muschinski, A., 1996: A similarity theory of locally homogenous and isotropic turbulence generated by a Smagorinsky-type LES. J. Fluid Mech., 325, 239260.

    • Search Google Scholar
    • Export Citation
  • National Science Foundation, 2007: Cyberinfrastructure vision for 21st century discovery. Tech. Rep. NSF 07–28, NSF Cyberinfrastructure Council. [Available online at http://www.nsf.gov/pubs/2007/nsf0728/index.jsp.]

    • Search Google Scholar
    • Export Citation
  • Nieuwstadt, F. T. M., , P. J. Mason, , C. H. Moeng, , and U. Schumann, 1993: Large-eddy simulation of the convective boundary layer: A comparison of four computer codes. Turbulent Shear Flows 8, F. Durst, Ed., Springer-Verlag, 343–367.

    • Search Google Scholar
    • Export Citation
  • Patton, E. G., , P. P. Sullivan, , and C.-H. Moeng, 2005: The influence of idealized heterogeneity on wet and dry planetary boundary layers coupled to the land surface. J. Atmos. Sci., 62, 20782097.

    • Search Google Scholar
    • Export Citation
  • Pekurovsky, D., , P. K. Yeung, , D. Donzis, , W. Pfeiffer, , and G. Chukkapallli, 2006: Scalability of a pseudospectral DNS turbulence code with 2D domain decomposition on Power4+/Federation and Blue Gene systems. ScicomP12 and SP-XXL, Boulder, CO, International Business Machines. [Available online at http://www.spscicomp.org/ScicomP12/Presentations/User/Pekurovsky.pdf.]

    • Search Google Scholar
    • Export Citation
  • Pope, S. B., 2000: Turbulent Flows. Cambridge University Press, 771 pp.

  • Raasch, S., , and M. Schröter, 2001: PALM—A large-eddy simulation model performing on massively parallel computers. Meteor. Z., 10, 363372.

    • Search Google Scholar
    • Export Citation
  • Schmidt, H., , and U. Schumann, 1989: Coherent structure of the convective boundary layer derived from large-eddy simulations. J. Fluid Mech., 200, 511562.

    • Search Google Scholar
    • Export Citation
  • Scotti, A., , C. Meneveau, , and D. K. Lilly, 1993: Generalized Smagorinsky model for anisotropic grids. Phys. Fluids A, 5, 23062308.

  • Silva Lopes, A., , and J. M. L. M. Palma, 2002: Numerical simulation of isotropic turbulence using a collocated approach and a nonorthogonal grid system. J. Comput. Phys., 175, 713738.

    • Search Google Scholar
    • Export Citation
  • Spalart, P. R., , R. D. Moser, , and M. M. Rogers, 1991: Spectral methods for the Navier–Stokes equations with one infinite and two periodic directions. J. Comput. Phys., 96, 297324.

    • Search Google Scholar
    • Export Citation
  • Sreenivasan, K. R., , A. Bershadskii, , and J. J. Niemela, 2002: Mean wind and its reversal in thermal convection. Phys. Rev. E, 65, 056306, doi:10.1103/PhysRevE.65.056306.

    • Search Google Scholar
    • Export Citation
  • Stevens, B., and Coauthors, 2005: Evaluation of large-eddy simulations via observations of nocturnal marine stratocumulus. Mon. Wea. Rev., 133, 14431462.

    • Search Google Scholar
    • Export Citation
  • Sullivan, P. P., , and J. C. McWilliams, 2010: Dynamics of winds and currents coupled to surface waves. Annu. Rev. Fluid Mech., 42, 1942.

    • Search Google Scholar
    • Export Citation
  • Sullivan, P. P., , J. C. McWilliams, , and C.-H. Moeng, 1994: A subgrid-scale model for large-eddy simulation of planetary boundary-layer flows. Bound.-Layer Meteor., 71, 247276.

    • Search Google Scholar
    • Export Citation
  • Sullivan, P. P., , J. C. McWilliams, , and C.-H. Moeng, 1996: A grid nesting method for large-eddy simulation of planetary boundary layer flows. Bound.-Layer Meteor., 80, 167202.

    • Search Google Scholar
    • Export Citation
  • Sullivan, P. P., , C.-H. Moeng, , B. Stevens, , D. H. Lenschow, , and S. D. Mayor, 1998: Structure of the entrainment zone capping the convective atmospheric boundary layer. J. Atmos. Sci., 55, 30423064.

    • Search Google Scholar
    • Export Citation
  • Sullivan, P. P., , T. W. Horst, , D. H. Lenschow, , C.-H. Moeng, , and J. C. Weil, 2003: Structure of subfilter-scale fluxes in the atmospheric surface layer with application to large-eddy simulation modeling. J. Fluid Mech., 482, 101139.

    • Search Google Scholar
    • Export Citation
  • Sullivan, P. P., , J. C. McWilliams, , and W. K. Melville, 2007: Surface gravity wave effects in the oceanic boundary layer: Large-eddy simulation with vortex force and stochastic breakers. J. Fluid Mech., 593, 405452.

    • Search Google Scholar
    • Export Citation
  • Sullivan, P. P., , J. B. Edson, , T. Hristov, , and J. C. McWilliams, 2008: Large-eddy simulations and observations of atmospheric marine boundary layers above non-equilibrium surface waves. J. Atmos. Sci., 65, 12251245.

    • Search Google Scholar
    • Export Citation
  • Tong, C., , J. C. Wyngaard, , S. Khanna, , and J. G. Brasseur, 1998: Resolvable- and subgrid-scale measurement in the atmospheric surface layer: Technique and issues. J. Atmos. Sci., 55, 31143126.

    • Search Google Scholar
    • Export Citation
  • Townsend, A. A., 1976: The Structure of Turbulent Shear Flow. Cambridge University Press, 429 pp.

  • Weil, J. C., 1988: Dispersion in the convective boundary layer. Lectures on Air Pollution Modeling, A. Venkatram and J. Wyngaard, Eds., Amer. Meteor. Soc., 167–227.

    • Search Google Scholar
    • Export Citation
  • Weil, J. C., 1990: A diagnosis of the asymmetry in top-down and bottom-up diffusion in a Lagrangian stochastic model. J. Atmos. Sci., 47, 501515.

    • Search Google Scholar
    • Export Citation
  • Werne, J., , and D. C. Fritts, 1999: Stratified shear turbulence: Evolution and statistics. Geophys. Res. Lett., 26, 439442.

  • Wyngaard, J. C., 1998: Boundary-layer modeling: History, philosophy, and sociology. Clear and Cloudy Boundary Layers, A. A. M. Holtslag and P. G. Duynkerke, Eds., Royal Netherlands Academy of Arts and Sciences, 325–332.

    • Search Google Scholar
    • Export Citation
  • Wyngaard, J. C., 2004a: Changing the face of small-scale meteorology. Atmospheric Turbulence and Mesoscale Meteorology, E. Federovich, R. Rotunno, and B. Stevens, Eds., Cambridge University Press, 17–34.

    • Search Google Scholar
    • Export Citation
  • Wyngaard, J. C., 2004b: Toward numerical modeling in the “terra incognita.” J. Atmos. Sci., 61, 18161826.

  • Wyngaard, J. C., 2010: Turbulence in the Atmosphere. Cambridge University Press, 393 pp.

  • View in gallery

    Variation of large-eddy Reynolds number Re with mesh resolution at heights z/zi = 0.1, 0.5, and 0.9 denoted by symbols □, ⋄, and ○, respectively; Re is computed from (6). Inertial subrange scaling is obeyed when the solid line becomes flat. Note the bottom and top x axes show the resolution ratios of zi to Δf and to CsΔf, respectively.

  • View in gallery

    Vertical profile of virtual potential temperature for varying mesh resolution. Note all simulations are started with the same three-layer structure for virtual potential temperature θi, indicated by the dotted line.

  • View in gallery

    Vertical profile of total temperature flux for varying mesh resolution.

  • View in gallery

    Variation of the boundary layer height zi with nondimensional time t/T; the large-eddy time scale T = zi/w*. The labels A–F correspond to the grid resolutions 323, 643, 1283, 2563, 5123, and 10243, respectively. The high-resolution runs (D–F) overlap. The simulation marked with an open square uses a mesh of 643 but with no monotone vertical temperature flux; that marked with an open circle uses a mesh of 2563 and is identical to simulation D but uses a filter width Δf equal to simulation B.

  • View in gallery

    Effect of mesh resolution on the (left) total turbulent kinetic energy (TKE) and (right) total temperature variance Θ2. TKE is normalized by and the temperature variance is normalized by Θ* = Q*/w*.

  • View in gallery

    Total variance (resolved plus SGS contributions) of the (left) vertical and (right) horizontal velocities. The horizontal variance uh is the sum of the u and υ components; see definition below (9).

  • View in gallery

    Two-dimensional energy spectra of (left) vertical velocity w and (right) horizontal velocity u in the PBL for varying meshes. The spectra are functions of the magnitude of the horizontal wavenumber vector kh = |k|. The groups of spectra at the top, middle, and bottom in each plot correspond to the heights z/zi = 0.9, 0.5, and 0.1, respectively. For clarity, the spectral amplitudes in each group are multiplied by the numerical factor on the left-hand side of the plot. The dashed line has slope .

  • View in gallery

    Two-dimensional energy spectrum of (left) vertical velocity w and (right) horizontal velocity u near the lower boundary at various heights z/zi = 0.1, 0.2, 0.3, and 0.5 for a simulations with 10243 grid points. The dashed line has slope .

  • View in gallery

    Effect of mesh resolution on resolved vertical velocity skewness . The lines legend indicates the mesh size of the various simulations. The skewness is computed using the resolved (or filtered) vertical velocity field . The observations are taken from the results provided in Moeng and Rotunno (1990).

  • View in gallery

    (a) Skewness from the 10243 simulation (solid), the 10243 simulation filtered in horizontal planes to 642 resolution (dotted), and the 643 simulation (dashed). (b) Third- and second-order SGS moments computed from 10243 simulation, showing , the third-order moment (dotted); , the second-order moment (dashed); and the SGS skewness correction [which appears in (16)] (solid).

  • View in gallery

    Comparison of third- and second-order resolved vertical velocity moments from the 10243 simulation (solid), the 10243 simulation filtered in horizontal planes to 642 resolution (dotted), and the 643 simulation (dashed), showing (a) normalized and (b) normalized .

  • View in gallery

    Effect of mesh resolution on resolved third-order moments (left) and (right) .

  • View in gallery

    Visualization of the vertical velocity field in a convective PBL at different heights from the 10243 simulation: z/zi = (top left) 0.04, (top right) 0.1, (bottom left) 0.5, and (bottom right) 0.9. The gray scale color bar changes between the panels and is in units of m s−1.

  • View in gallery

    Visualization of 105 particles randomly released in a convective PBL at z/zi ~ 0.01 over a limited horizontal extent from the 10243 simulation of convection. The viewed area, ~3.8% of the total horizontal domain, is the topmost left corner from the top-left panel of Fig. 13. Notice the evolution of the larger-scale line of convection into small-scale vortical motions that resemble dust devils. Time advances from left to right beginning along the top row of images. The images are 71.6, 151, and 390 s after the initial release. Vertical vorticity is shown in the bottom-right panel at t = 390 s. The gray scale bar is in units of s−1.

  • View in gallery

    The 2D domain decomposition on nine processors: (a) base state with y–z decomposition, (b) x–z decomposition used for computation of y derivatives and 2D planar FFT, and (c) x–y decomposition used in the tridiagonal matrix inversion of the pressure Poisson equation.

  • View in gallery

    Computational time per grid point for different combinations of problem size and 2D domain decomposition for the Cray XT4 (an example of strong scaling), showing problem sizes (a) 5123 (⋄), (b) 10243 (○), (c) 20483 (□), and (d) 30723 (Δ). For a given number of total processors NP the symbols are varying vertical and horizontal decompositions [i.e., different combinations (NPz, NPxy)].

  • View in gallery

    Computational time per grid point for a fixed amount of work per processor (an example of weak scaling). Shown are 60 000 points per processor for the Cray XT4 (○), dual core IBM SP5+ (⋄), and single core IBM SP5 (□), and 524 288 points per processor for the Cray XT4 (Δ). For a fixed number of total processors NP multiple symbols are different combinations of (NPz, NPxy).

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 137 137 60
PDF Downloads 119 119 56

The Effect of Mesh Resolution on Convective Boundary Layer Statistics and Structures Generated by Large-Eddy Simulation

View More View Less
  • 1 National Center for Atmospheric Research, Boulder, Colorado
© Get Permissions
Full access

Abstract

A massively parallel large-eddy simulation (LES) code for planetary boundary layers (PBLs) that utilizes pseudospectral differencing in horizontal planes and solves an elliptic pressure equation is described. As an application, this code is used to examine the numerical convergence of the three-dimensional time-dependent simulations of a weakly sheared daytime convective PBL on meshes varying from 323 to 10243 grid points. Based on the variation of the second-order statistics, energy spectra, and entrainment statistics, LES solutions converge provided there is adequate separation between the energy-containing eddies and those near the filter cutoff scale. For the convective PBL studied, the majority of the low-order moment statistics (means, variances, and fluxes) become grid independent when the ratio zi/(CsΔf) > 310, where zi is the boundary layer height, Δf is the filter cutoff scale, and Cs is the Smagorinsky constant. In this regime, the spectra show clear Kolmogorov inertial subrange scaling. The bulk entrainment rate determined from the time variation of the boundary layer height we = dzi/dt is a sensitive measure of the LES solution convergence; we becomes grid independent when the vertical grid resolution is able to capture both the mean structure of the overlying inversion and the turbulence. For all mesh resolutions used, the vertical temperature flux profile varies linearly over the interior of the boundary layer and the minimum temperature flux is approximately −0.2 of the surface heat flux. Thus, these metrics are inadequate measures of solution convergence. The variation of the vertical velocity skewness and third-order moments expose the LES’s sensitivity to grid resolution.

Corresponding author address: Peter P. Sullivan, MMM Division, NCAR, Boulder, CO 80307–3000. E-mail: pps@ucar.edu

Abstract

A massively parallel large-eddy simulation (LES) code for planetary boundary layers (PBLs) that utilizes pseudospectral differencing in horizontal planes and solves an elliptic pressure equation is described. As an application, this code is used to examine the numerical convergence of the three-dimensional time-dependent simulations of a weakly sheared daytime convective PBL on meshes varying from 323 to 10243 grid points. Based on the variation of the second-order statistics, energy spectra, and entrainment statistics, LES solutions converge provided there is adequate separation between the energy-containing eddies and those near the filter cutoff scale. For the convective PBL studied, the majority of the low-order moment statistics (means, variances, and fluxes) become grid independent when the ratio zi/(CsΔf) > 310, where zi is the boundary layer height, Δf is the filter cutoff scale, and Cs is the Smagorinsky constant. In this regime, the spectra show clear Kolmogorov inertial subrange scaling. The bulk entrainment rate determined from the time variation of the boundary layer height we = dzi/dt is a sensitive measure of the LES solution convergence; we becomes grid independent when the vertical grid resolution is able to capture both the mean structure of the overlying inversion and the turbulence. For all mesh resolutions used, the vertical temperature flux profile varies linearly over the interior of the boundary layer and the minimum temperature flux is approximately −0.2 of the surface heat flux. Thus, these metrics are inadequate measures of solution convergence. The variation of the vertical velocity skewness and third-order moments expose the LES’s sensitivity to grid resolution.

Corresponding author address: Peter P. Sullivan, MMM Division, NCAR, Boulder, CO 80307–3000. E-mail: pps@ucar.edu
Keywords: Boundary layer

1. Introduction

Large-scale parallel computing has the potential to alter the landscape of turbulence simulations in the atmospheric and oceanic planetary boundary layers (PBLs) as increased computer power using O(104–105) or more processors (National Science Foundation 2007) will permit large-eddy simulations (LESs) of turbulent PBLs coupling small and large scales in realistic outdoor environments. Applications include, atmosphere–land interactions (Patton et al. 2005), boundary layers with surface water wave effects (Sullivan and McWilliams 2010; Sullivan et al. 2007, 2008), weakly stable nocturnal flows (Beare et al. 2006), flow in complex terrain (Lundquist et al. 2010), stratocumulus clouds (Stevens et al. 2005), tropical boundary layers beneath deep convection (Moeng et al. 2009), and coupling with mesoscale weather events (Bryan et al. 2003), to mention just a few.

Given the prominent and important role of LES in studying boundary layer dynamics (Wyngaard 1998), it is important to examine the quality of LES solutions, and in particular their dependence on the grid mesh, subgrid-scale (SGS) parameterizations, numerical discretizations, and surface boundary conditions. Assessing the numerical convergence and the quantification of uncertainty in LES, induced by modeling and numerical errors, is compounded by the significant computational expense needed to carry out meaningful grid refinement for a three-dimensional time-dependent turbulent flow (Pope 2000). The subgrid-scale model and numerical discretization errors are intertwined since both depend explicitly on the mesh spacing (Chow and Moin 2003; Meyers et al. 2007; Geurts and Fröhlich 2002). The effective Reynolds number associated with the subgrid-scale model can vary widely so that LES solutions can be either deterministic or stochastic (Bryan et al. 2003; Wyngaard 2004a). When the effective Reynolds number is sufficiently large, resolved turbulence is supported and LES solutions are stochastic, which requires that time- and space-averaged statistics be examined in order to judge convergence. Designing metrics to assess solution error is not obvious (Celik et al. 2006). Meyers et al. (2007) propose a framework for LES model evaluation using large- and small-scale metrics that are both physics and mathematics based. They are able to extract LES discretization errors for idealized homogeneous isotropic turbulence simulations with the Smagorinsky model but rely on a direct numerical simulation (DNS) as ground truth in their evaluations, which is not available for the high-Reynolds number PBL.

Here, we investigate one aspect of assessing the quality of LES solutions, namely the sensitivity and convergence of LES solutions as the grid mesh is substantially varied for a particular choice of subgrid-scale model. The physical problem investigated is a very weakly sheared daytime convective PBL similar to that studied by Schmidt and Schumann (1989). There have been a few previous investigations that explored some aspects of the convergence of LES solutions mainly focused on an intercomparison of different codes on a similar mesh [e.g., see LES intercomparison studies by Beare et al. (2006), Stevens et al. (2005), Bretherton et al. (1999), Andren et al. (1994), Nieuwstadt et al. (1993) and Fedorovich et al. (2004)]. Bryan et al. (2003) examined the resolution requirements to simulate convective weather events and found that the statistical properties of squall lines are still not converged with a grid spacing of 125 m. Past investigations have been carried out with the intent of clarifying the behavior of LES for different PBL flows. Nieuwstadt et al. (1993) reports on the first intercomparison of simulation codes for the convective PBL using coarse 403 meshes. Andren et al. (1994) examined neutrally stratified PBLs, Beare et al. (2006) considered the behavior of the stable PBL, and Bretherton et al. (1999) studied radiatively driven entrainment in a smoke cloud. Previous work aligned with the present study is documented by Mason and Brown (1999). They examined a modest range of domain size, grid resolutions, and subgrid-scale model constants but were particularly interested in the influence of filter-scale CsΔf; Cs is the Smagorinsky constant and Δf is a characteristic subgrid length scale.

The outline of the paper is as follows: section 2 is a brief introduction to the LES equations appropriate for a high-Reynolds number PBL; section 3 describes the LES grid refinement experiments; results are presented in section 4; section 5 provides a summary of the findings; and the appendix provides technical details about the LES code parallelization and performance.

2. LES equations

Typical LES model equations for a dry atmospheric planetary boundary layer under the Boussinesq approximation include (a) transport equations for momentum, (b) a transport equation for a conserved buoyancy variable (e.g., virtual potential temperature ), and (c) a discrete Poisson equation for a pressure variable to enforce incompressibility, as well as closure expressions for subgrid-scale variables [e.g., an SGS equation for turbulent kinetic energy (TKE) e]. Formally, the LES equations are derived by applying a low-pass spatial filter to the equations of motion that leads to the decomposition of the total velocity and total virtual potential temperature , where denote resolved and subgrid fields, respectively. The LES model equations used are (Moeng 1984; Moeng and Sullivan 2002)
e1a
e1b
e1c
The pressure needed to make the flow divergence free is the solution of the Poisson equation
e2
Equation (2) is formed by applying the discrete divergence operator to (1a) and collecting all terms except the pressure in the source term r [see Sullivan et al. (1996) for details].
In (1), , where f is the Coriolis parameter and is the unit vector in the z direction, Ug is the geostrophic wind with x, y components (Ug, Vg), and β = g/θ0 is the buoyancy parameter with (g, θ0) denoting gravity and a reference virtual potential temperature, respectively. The SGS momentum and scalar fluxes and SGS energy are
e3a
e3b
e3c
In the SGS TKE equation (1c) terms on the right side are subgrid-scale production and buoyancy , diffusion , and dissipation . The modeling of these terms and the SGS fluxes in terms of an eddy viscosity νt and diffusivity νH are fully described in Deardorff (1980), Moeng (1984), Moeng and Wyngaard (1988), and Sullivan et al. (1994). An excellent and insightful discussion of the subgrid-scale dynamics contained in (1c) is given by Moeng and Wyngaard (1988, 3581–3585). We are aware that the specification of the subgrid-scale fluxes using a TKE eddy viscosity model is one of many proposals available in the literature (see, e.g., Meneveau and Katz 2000; Geurts 2001; Sullivan et al. 2003; Wyngaard 2004b). However, the objective here is not to focus on the impact of different SGS prescriptions but rather to examine the solution mesh dependence given a particular choice of SGS.

An important difference between smooth and rough wall LES is the specification of surface boundary conditions. As is common practice with geophysical flows, we impose rough wall boundary conditions based on a drag rule where the surface transfer coefficients are determined from Monin–Obukhov similarity functions (Moeng 1984; Moeng and Sullivan 1994). A high Reynolds number model for viscous dissipation is used in (1c) [see discussion near (6)]. Thus, molecular viscosity and diffusivity do not appear in the LES equation set. The sidewall (x, y) boundary conditions are periodic and a radiation boundary condition (Klemp and Durran 1983) is used at the top of the domain.

In our LES code, (1) are integrated in time using a fractional step method. The spatial discretization is second-order finite difference in the vertical direction and pseudospectral in the horizontal planes. The resolved vertical flux in (1b) receives special treatment. It is determined using a second-order near monotone scheme described by Beets and Koren (1996) (see also Koren 1993). This upwind scheme prevents unrealizable oscillations in regions of strong vertical scalar gradients. Dynamic time stepping utilizing a third-order Runge–Kutta scheme with a fixed Courant–Fredrichs–Lewy (CFL) number (Sullivan et al. 1996; Spalart et al. 1991) is employed. Evaluating horizontal derivatives with fast Fourier transforms (FFTs) and solving the elliptic pressure equation are nonlocal operations that impact the code parallelization (see the appendix).

3. Design of LES experiments

A suite of simulations on a fixed computational domain with varying grid resolutions is performed to examine the convergence of the LES equations given in section 2 using the parallel algorithm described in the appendix. A canonical daytime convective PBL is simulated in a computational domain (Lx, Ly, Lz) = (5120, 5120, 2048) m. Six simulations are performed with grid meshes of 323, 643, 1283, 2563, 5123, and 10243, and for each mesh the spacing is held constant in the three (x, y, z) directions (see Table 1). The PBL is driven by a constant surface buoyancy flux Q* = 0.24 K m s−1 and weak geostrophic winds (Ug, Vg) = (1, 0) m s−1. Other external inputs are surface roughness z0 = 0.1 m, Coriolis parameter f = 1 × 10−4 s−1, and initial inversion height zi ~1024 m. In terms of the initial PBL height, the computational domain is (Lx, Ly, Lz)/zi = (5, 5, 2), which is sufficient to allow fully turbulent flow fields to develop independently of the periodic sidewall boundary conditions (e.g., Schmidt and Schumann 1989). At long time scales (t ≥ 8 h) the horizontal domain should be expanded to accommodate the very large structures that can develop under persistent forcing, as discovered by Jonker et al. (1999) and de Roode et al. (2004).

Table 1.

Simulation grid spacings.

Table 1.

The initial sounding of virtual potential temperature (to streamline the text, hereafter virtual potential temperature θ is simply referred to as “temperature”) has a three-layer structure:
e4
Thus, a sharp jump in temperature of 8 K is imposed over a depth of 100 m near the top of the PBL. For this combination of geostrophic wind and surface buoyancy flux the Monin–Obukhov length scale L ≈ −1.5 m and thus the PBL is dominated by convective forcing since −zi/L = O(500). All simulations are started from small random seed perturbations in temperature near the surface. The simulations are carried forward for about 25 large eddy turnover times T = zi/w*, where the Deardorff convective velocity scale w* = (gQ*zi/θ0)1/3. At each time step, the boundary layer top zi is diagnosed using the “maximum gradient method” (Sullivan et al. 1998). Statistics are generated by averaging in horizontal x–y planes and over the time interval 10T–25T; these averages, which approximate the ensemble average, are indicated by 〈·〉. Also, we often compute statistics of a resolved turbulence fluctuation . In the case of vertical velocity, , since horizontal periodic boundary conditions and no flow through the bottom boundary require in order to satisfy (2).

Grid resolution tests with LES are demanding since the resolved turbulent motions are always 3D and time dependent. For rough-wall LES of a given domain size, the number of mesh points in a single direction N ~ (Lxx) and hence N3 ~ (Lxx)3, assuming equal spacing in all three directions. However, refining the mesh also lowers the acceptable time step owing to the limits imposed by a CFL constraint; that is, CFL = |u|max Δtx. Thus, as the grid spacing decreases, the number of time steps needed to advance the solutions to the same time further increases by the factor M ~ Lxx (see, e.g., Pope 2000, p. 348). The total computational work for a complete simulation is then M · N3 ~ (Lxx)4. As an example of the steep climb in work with increasing resolution, the computational effort on a mesh with 10243 grid points is approximately 4096 times greater than the work required on a mesh with 1283 grid points. This underestimates the effort by a factor of 2 since our computations are dominated by FFT work, which scales as NlogN in both x and y.

4. Results

In the analysis of the LES solutions we discuss the variation of statistics and vertical profiles as a function of the mesh resolution ratio zif or ziz; here zi is the PBL depth and Δf is the LES filter width, which is related to the mesh spacings Δxi, as discussed below. In the interior of the PBL, away from the surface layer and entrainment zone, numerous observational and LES studies find that zi is a characteristic scale of the energy containing eddies in the convective PBL (e.g., Deardorff 1972a; Lenschow et al. 1980; Lothon et al. 2009; Jonker et al. 1999). Thus, the nondimensional ratio zif can be interpreted as a measure of the scale separation between the energy-containing eddies and those near the filter cutoff. When the SGS closure is the Smagorinsky model, Mason and Brown (1999) and Pope (2000) prefer to interpret the LES set of equations as a numerical system with the degrees of freedom limited by a low-pass “Smagorinsky filter.” The cutoff scale of the filter is CsΔf, with Cs equal to the Smagorinsky constant. Muschinski (1996) builds on this interpretation and discusses the properties of a non-Newtonian LES fluid with a Smagorinsky viscosity. To place our simulations in the context of this alternate interpretation, we also present the results as a function of the resolution ratio zi/(CsΔf). In either interpretation, when zif ≫ 1 LES solutions have a wide separation between the energy-containing eddies and those near the filter cutoff scale. Observations of subgrid-scale turbulence in the atmospheric surface layer demonstrate that a similar ratio of scales Λwf, where Λw is the scale of the peak in the vertical velocity spectrum, is a useful dimensionless parameter that collapses the variation of subgrid-scale turbulence over a range of stratification and filter widths (Sullivan et al. 2003).

A summary of bulk PBL properties generated from the various simulations is provided in Table 2. Entries in this table are PBL depth zi, convective velocity scale w*, normalized entrainment rate ratio we/w*, large-eddy Reynolds number at mid-PBL Re, friction velocity ratio u*/w*, bottom and top of the entrainment zone (δb, δt)/zi, and the ratio of PBL depth to filter width and vertical resolution zi/(Δf, CsΔf, Δz). Note that δb and δt are the endpoints of the entrainment zone defined as the region where the total vertical temperature flux is negative. A broad look at the tabulated results shows that w* is almost invariant with the mesh resolution, while the friction velocity shows a slight downward trend of ~10% as the mesh varies. Our values of u*/w* ~ 0.08 for zi/z0 ~ 104 are close to those predicted by Schmidt and Schumann (1989). Meanwhile, the entrainment rate and entrainment zone depth vary substantially on the coarser meshes. The variations of the bulk properties and the vertical profiles of selected flow variables are discussed below.

Table 2.

Bulk simulation properties.

Table 2.

a. Inertial subrange scaling

A fundamental basis of high Reynolds number LES is that the resolved (large-eddy) turbulence is independent of subgrid-scale influences, namely the SGS viscosity νt and scalar diffusivity νH. In other words, the large-eddy Reynolds number Re in LES solutions must be sufficiently large that the resolved flow is in a regime of so-called “Reynolds number similarity” (Townsend 1976; Wyngaard 2010). We follow Moeng and Wyngaard (1988) and define Re for an LES of a convective PBL based on the SGS viscosity νt and characteristic velocity and length scales (u, ℓ) = (w*, zi). Thus,
e5
where we adopt the definition of SGS viscosity . Here Ck = 0.1 is a modeling constant that follows from matching with an inertial subrange spectrum (Moeng and Wyngaard 1988). In our pseudospectral code, the filter width is the characteristic length scale of the cell averaging volume , where the grid spacings are Δxi and are increased by a factor of 3/2 to account for dealiasing of the upper ⅓ wavenumbers. Inserting the definition of w* into (5) and adopting the high Reynolds number inertial subrange dissipation model (Lilly 1967; Moeng and Wyngaard 1988)
e6
leads to
e7
In (6) the modeling constant and for a production–dissipation balance in (1c) the Smagorinsky constant . Equation (7) exposes the dependence on the Smagorinsky filter scale CsΔf. Moeng and Wyngaard (1988) further argue that if the filter cutoff lies in the inertial subrange, then the net dissipation should be constant, which leads to the scaling Re ~ (zif)4/3. This is similar to how the large-scale Reynolds number varies in direct numerical simulation (see Pope 2000, p. 347). For our LES experiments with different meshes we estimate based on (5) that at mid-PBL Re varies by almost two orders of magnitude, Re = [240, 20600] (see Table 2). To test the LES scaling suggested by (7) we show the product Ref/zi)4/3 at three heights z/zi = (0.1, 0.5, 0.9) for varying mesh resolution in Fig. 1. Note that Re is largest in the upper part of the PBL and smallest near the surface, which is a consequence of the SGS e dependence in (5); 〈e(z)〉 has a maximum near the surface and decreases monotonically toward the top of the PBL. Also, in the PBL interior the inertial range scaling suggested by (7) is indeed obeyed when zif > 60 or zi/(CsΔf) > 310; this corresponds to meshes greater than 2563 (Table 2). Moeng and Wyngaard (1988) comment that their 963 computations with Re = 1000 fall within the inertial subrange but are likely somewhat close in scale to the energy-containing eddies, which is confirmed by the present calculations. Of course, below z/zi < 0.1, Re is even smaller because of the high levels of SGS e and thus meshes finer than 2563 are needed close to the wall before simulations will be able to adequately reproduce an inertial subrange (see section 4d).
Fig. 1.
Fig. 1.

Variation of large-eddy Reynolds number Re with mesh resolution at heights z/zi = 0.1, 0.5, and 0.9 denoted by symbols □, ⋄, and ○, respectively; Re is computed from (6). Inertial subrange scaling is obeyed when the solid line becomes flat. Note the bottom and top x axes show the resolution ratios of zi to Δf and to CsΔf, respectively.

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

An alternate but equivalent statement of the high Reynolds number scaling Re ~ (zif)4/3 is that the dissipation is independent of the grid mesh. As a consistency check, we computed the time and horizontally averaged dissipation from (6) at heights z/zi = 0.1, 0.5, and 0.9 from the different simulations. We find that also tends to a constant grid-independent value in the interior of the PBL when the mesh is 2563 or greater. This shows that high Reynolds number LES obeys one of its fundamental assumptions as the mesh is refined.

b. Temperature profiles and entrainment statistics

The vertical structure of the mean temperature and its turbulent flux profile are dependent on the grid resolution as shown in Figs. 2 and 3. Figure 2 compares the temperature profiles at time t = 15T, which is well beyond the initial spinup period for the turbulence and is representative of the late-time quasi-steady behavior of . All profiles display a similar well-mixed structure over the interior of the boundary layer 0.1 < z/zi < 0.9. The profile of near the surface and in particular near zi exhibits a grid resolution sensitivity that impacts the interior temperatures. Since the surface heating is constant in time across the simulations, the increased warming observed in the mid-PBL with the lower-resolution simulations must result from an increase in entrainment, as discussed below. Recall that all simulations are initiated with the same three-layer temperature sounding (4); however, on the coarse meshes the temperature profile reaches a quasi-equilibrium state with a much weaker inversion.

Fig. 2.
Fig. 2.

Vertical profile of virtual potential temperature for varying mesh resolution. Note all simulations are started with the same three-layer structure for virtual potential temperature θi, indicated by the dotted line.

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

Fig. 3.
Fig. 3.

Vertical profile of total temperature flux for varying mesh resolution.

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

The response of the temperature flux profiles to the varying mean θ profiles, shown in Fig. 3, is interesting. Despite the radical changes to the overlying temperature structure with varying mesh, all the temperature flux profiles decrease linearly over the boundary layer, reaching a minimum (negative) value near and below zi. Note that Fig. 3 shows the total temperature flux (i.e., the sum of resolved plus subgrid-scale fluxes where the latter is retrieved from the SGS eddy viscosity model ). The minimum temperature flux is weakly dependent on the mesh resolution; it becomes slightly less negative as Δf decreases. However, the depth of the entrainment zone, defined as the layer where the temperature flux is less than zero, expands considerably as the mesh is coarsened. This is consistent with the observed changes in the mean temperature profiles.

The temporal variation of the boundary layer inversion height zi(t), shown in Fig. 4, is a strong measure of solution convergence. Here zi is determined using the maximum vertical gradient in temperature; that is, for each x, y gridpoint we search along a vertical column to find the location of the maximum in and then horizontally average all those positions to define zi at a particular t. This technique closely tracks local changes in the inversion (Sullivan et al. 1998; Davis et al. 2000). We notice immediately that the boundary layer in the low-resolution simulations entrains fluid much more rapidly than in the fine-mesh simulations, which reflects the weakened inversions discussed previously. A critical parameter, the entrainment rate we = dzi/dt, is then a function of the mesh resolution; we determined from a linear least squares curve fit to the variation of zi(t) over the interval 10T–25T is listed in Table 2. In the coarse 323 simulation the nondimensional entrainment rate we/w* ~ 9.2 × 10−3, which is more than 85% larger than the finest 10243 resolution run. Notice in Table 2 that the entrainment rate does not change appreciably once the mesh resolution exceeds 2563, while the entrainment rate from the 1283 simulation is about 30% larger than the average of the fine-mesh runs (runs D, E, and F).

Fig. 4.
Fig. 4.

Variation of the boundary layer height zi with nondimensional time t/T; the large-eddy time scale T = zi/w*. The labels A–F correspond to the grid resolutions 323, 643, 1283, 2563, 5123, and 10243, respectively. The high-resolution runs (D–F) overlap. The simulation marked with an open square uses a mesh of 643 but with no monotone vertical temperature flux; that marked with an open circle uses a mesh of 2563 and is identical to simulation D but uses a filter width Δf equal to simulation B.

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

To further expose the coupling between the mean temperature field and turbulence in the entrainment zone we examine the average budget equations for the resolved vertical temperature flux and temperature variance (e.g., see Mironov et al. 2000):
e8a
e8b
In these equations T is turbulent transport, M is mean-gradient production, B is buoyant production, P is pressure destruction, and S is a subgrid-scale term (i.e., a correlation between resolved and SGS variables). Temperature flux (8a) and temperature variance (8b) are coupled with each other and both depend on the mean temperature through its vertical gradient . There is of course feedback to the mean temperature through the vertical divergence of temperature flux. In (8a), the resolved vertical flux depends on the mean-gradient production term M, which acts as both source and sink since the mean temperature gradient has both positive and negative slopes in the convective PBL. In (8b), resolved temperature variance reflects the balance between flux mean-gradient production and the subgrid-scale term. On the coarse meshes the internal balance within the LES leads to weaker temperature gradients near zi, but the region where is spread over a greater extent of the PBL (see Fig. 2). As a result, the production of temperature variance by term M in (8b) on the coarse meshes is significantly lower since the minimum value of the temperature flux changes only slightly with resolution. At the same time, the mean-gradient term M in (8a) acts as a sink for temperature flux over a larger depth near the PBL inversion and hence the entrainment zone—that is, the layer where —expands on the coarse meshes, as observed in Fig. 3 (see also Table 2).

The couplings among mean temperature, temperature flux, and temperature variance in (8) are subtle and complex and apparently depend critically on the mean temperature gradient. This in turn impacts the overall entrainment predicted by LES. To illustrate the influence of on the LES solutions we perform two additional simulations. Simulation D1 uses a 2563 mesh but sets the filter width Δf equal to the value for the 643 mesh with all other parameters held constant. Thus D1 has fine vertical resolution Δz = 8 m but sets Δf = 77.2 m. Simulation B1 uses a 643 mesh but turns off the monotone computation of the vertical temperature flux in (1b). Figure 4 shows the entrainment rate D1 ≈ D and B1 ≈ B. These results indicate that the weakening of the inversion in the coarse-mesh simulations is a consequence of sparse vertical resolution of the mean temperature gradient and its internal couplings with the turbulence and is not a result of monotone numerics and/or SGS effects. We note that LES studies of stratocumulus-topped PBLs by Stevens et al. (2005), which have very sharp inversions, find that the best comparison between LES and observations occurs when the vertical resolution within the LES is very fine. The meshes used in those simulations are, however, very anisotropic, with (Δx, Δy) ≫ Δz.

Based on our LES experiments we conclude that to generate grid-independent solutions the mesh needs to have sufficiently fine vertical resolution to capture both the mean temperature gradients in the overlying inversion and the turbulence. However, vertical refinement requires a comparable refinement of the horizontal grid in order to maintain reasonable aspect ratio grids; grid isotropy impacts inertial range SGS constants (e.g., Scotti et al. 1993). Generally, the impact of grid anisotropy Δx ≠ Δy ≠ Δz on LES solutions is not well understood (e.g., Kaltenbach 1997; Silva Lopes and Palma 2002). We note, however, that in all our computations Δx = Δy and hence the explicit (dealiasing) filtering used in horizontal x–y planes is isotropic. Tong et al. (1998) shows that 2D (isotropic) filtering, as used here, is nearly equivalent to 3D filtering.

These mesh resolution experiments have implications for LES studies of entrainment. There is a subtle interplay among mesh resolution, the overlying inversion, the minimum temperature flux, and the entrainment rate. Insufficient vertical resolution weakens the inversion and increases the entrainment rate while maintaining nearly the same minimum temperature flux. A first-order entrainment jump model (Betts 1974) shows how a finite inversion thickness contributes to the entrainment rate (see Sullivan et al. 1998). Linearity of the temperature (or heat) flux profile and minimum temperature flux approximately equal to −0.2Q* are relatively insensitive to the mesh resolution and thus are insufficient to judge the convergence of LES solutions for the convective boundary layer. The variation of the entrainment rate we = dzi/dt is a much more sensitive indicator of LES solution convergence.

c. Convergence of variances statistics

One of the main attributes of LES is its ability to accurately compute turbulence. To judge whether the turbulence statistics generated by LES are mesh invariant, we need to account for both the resolved and SGS contributions to the variances. Figure 5 compares vertical profiles of the normalized total TKE
e9
for varying mesh resolutions. Ideally, with small sampling errors all profiles should collapse onto the same curve independent of Δf. Overall there is broad agreement among the profile shapes for varying Δf, but the TKE profile displays more sampling variability than the total temperature flux, especially in mid-PBL. Despite this variability, the shape of the TKE profile near zi is clearly mesh dependent, which is an indirect consequence of the changes to the temperature structure in the entrainment zone discussed in section 4b. Also, there is a persistent trend where the total TKE in the lower PBL z/zi < 0.5 on the coarse meshes is reduced compared to the fine mesh calculations. Near the surface the total TKE profiles on the coarse meshes show noticeable departures from their fine mesh counterparts. This is likely due to a combination of effects such as inaccurate modeling of SGS fluxes near a rough boundary (e.g., Sullivan et al. 2003; Brasseur and Wei 2010) and interactions with an outer flow that varies with mesh resolution. The large entrainment rates on the coarse meshes imply that the turbulence in those PBLs is only quasi-stationary in time.
Fig. 5.
Fig. 5.

Effect of mesh resolution on the (left) total turbulent kinetic energy (TKE) and (right) total temperature variance Θ2. TKE is normalized by and the temperature variance is normalized by Θ* = Q*/w*.

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

Inspection of the vertical profiles of total vertical variance and total horizontal velocity variance (see Fig. 6) provides additional insight as to how the TKE varies with zi/Δf. The value 2e/3 is an estimate of the contribution from the SGS velocity variance for an isotropic SGS model and thus . All the profiles of the total w variance display the same smooth slightly asymmetric shape over the PBL. There is a remarkably good collapse among the profiles when the mesh is 2563 [i.e., zi/(CsΔf) > 310]. The variability in the TKE profile observed in Fig. 5 clearly arises from the horizontal variances. The variance has peaks near the ground and zi; the upper maximum shifts its vertical location and shape depending on the structure of the overlying inversion. Generally, we do not see the same high degree of convergence of the horizontal variance with mesh resolution as observed for the vertical velocity variance. A possible cause of this variability is the presence of abrupt wind reversals that occur sporadically over long time intervals in thermal convection as observed by Sreenivasan et al. (2002).

Fig. 6.
Fig. 6.

Total variance (resolved plus SGS contributions) of the (left) vertical and (right) horizontal velocities. The horizontal variance uh is the sum of the u and υ components; see definition below (9).

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

Our SGS closure prescription for temperature flux is based on an eddy viscosity and a turbulent Prandtl number assumption: we do not carry a prognostic equation for the subgrid-scale temperature variance. The total temperature variance is
e10
and we diagnose the SGS variance contribution from
e11
Equation (11) is a truncated balance between variance destruction and variance production by the vertical temperature gradient. Here is the SGS vertical temperature flux in (3b), θ* = Q*/w*, Cθ = 2.02 (Schmidt and Schumann 1989; Deardorff 1972b), and ℓf is a stability-corrected length scale (Moeng 1984; Deardorff 1980). We find that only very near the surface. In the inversion ℓf is small because of the stable stratification that damps the SGS temperature variance in (11). Figure 5 shows the total temperature variance Θ2(z) over the entire PBL. In the interior 0.1 < z/zi < 0.9, the temperature variance on all meshes is small and appears converged. At the upper edge of the PBL the temperature variance has a pronounced maximum that varies with mesh resolution. The peak total variance on the 10243 mesh is nearly 5 times as large as on the 643 mesh and furthermore is concentrated over a thinner vertical extent. The weaker temperature gradient that develops on the coarse mesh, discussed in section 4b, greatly reduces the temperature variance. Temperature variance generated by LES converges in the interior of the PBL but becomes mesh dependent in the entrainment zone and very near the surface because of the variation of the mean temperature gradient and a mesh-dependent triple-moment term shown in section 4e. SGS models based on rate equations that include a prognostic conservation equation for SGS variance should improve the convergence of the LES solutions for temperature variance in these regions (Wyngaard 2004b; Hatlee and Wyngaard 2007).

d. Spectral analysis

Figure 7 shows two-dimensional spectra of the vertical and horizontal velocity at nondimensional heights z/zi = (0.9, 0.5, 0.1) for varying mesh resolutions. These spectra are functions of the horizontal wavenumber vector and are smoothed by averaging in circular rings at constant and over time. These two-dimensional spectra vanish at zero wavenumber and are more representative of the spatial eddy scale than their one-dimensional counterparts, which are contaminated by aliasing from averaging in either x or y directions (see Wyngaard 2010, p. 351).

Fig. 7.
Fig. 7.

Two-dimensional energy spectra of (left) vertical velocity w and (right) horizontal velocity u in the PBL for varying meshes. The spectra are functions of the magnitude of the horizontal wavenumber vector kh = |k|. The groups of spectra at the top, middle, and bottom in each plot correspond to the heights z/zi = 0.9, 0.5, and 0.1, respectively. For clarity, the spectral amplitudes in each group are multiplied by the numerical factor on the left-hand side of the plot. The dashed line has slope .

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

In the upper boundary layer, z/zi = 0.9, all the meshes capture the peak in the vertical velocity spectrum reasonably well and also display a slope, which is especially clear in the fine 10243 resolution run. There is a small departure from at the highest wavenumbers in the spectrum of the horizontal velocity. Similar behavior occurs at mid-PBL z/zi = 0.5, as shown in Fig. 7. Near the outer edge of the surface layer (z/zi = 0.1), we notice a pronounced broadening of the peak in the vertical velocity spectrum with a clear shift to higher wavenumbers; this is due to inviscid blocking by the presence of the wall. The coarser-resolution runs with meshes of 1283 and less are just barely able to resolve the peak in the vertical velocity spectrum at this height. It is encouraging that all runs display a similar variation at low wavenumbers khzi ≤ 10.

The spectrum of horizontal velocity displays an intriguing behavior at z/zi = 0.1, and to a lesser extent at z/zi = 0.9, Its peak energy is clearly at a lower wavenumber compared to the vertical velocity, and the finest-resolution run hints at a two-slope character (i.e., it displays a slope transition near khzi ~ 25). This behavior reflects the redistribution of energy near the lower surface because of the wall presence. This is exposed more clearly in Fig. 8 where we show the z variation of the spectra from the 10243 simulation as the lower boundary is approached. We notice a smooth gradual decrease in the magnitude of the vertical velocity spectrum at low wavenumbers accompanied by a gradual shift in the peak toward higher wavenumbers as z/zi decreases. A slope of is always maintained as z/zi decreases as predicted by Wyngaard (2010, p. 355). The spectrum of the horizontal velocity, however, displays an opposite trend that reflects the amplification of the u variance as z decreases. We notice that the increases in u variance occur at low wavenumbers with the peak in its spectrum growing by almost a factor of 10 at wavenumber khzi ~ 3, with almost no change to the spectral components khzi ≥ 40. These changes are barely captured by the 1283 simulation. Near the lower boundary, the spectral distribution of energy reflects the enhancement of the (u, υ) variances caused by descending downdrafts that transfer energy into horizontal motions. Pope (2000, p. 433) discusses a number of different effects induced by the presence of a wall.

Fig. 8.
Fig. 8.

Two-dimensional energy spectrum of (left) vertical velocity w and (right) horizontal velocity u near the lower boundary at various heights z/zi = 0.1, 0.2, 0.3, and 0.5 for a simulations with 10243 grid points. The dashed line has slope .

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

e. High-order moments

Velocity and scalar moments higher than second order appear in ensemble average TKE and flux budgets and are used in the interpretation of PBL dynamics (e.g., Mironov 2009). Often LES flow fields are used to compute high-order moments, but it is unknown how grid resolution impacts these estimates. Moeng and Rotunno (1990) identify the vertical velocity skewness Sw as a critical parameter in boundary layer dynamics. In convective PBLs, Sw is an indicator of the updraft–downdraft distribution, provides clues about vertical transport, and is utilized in dispersion studies (Weil 1988, 1990). Further, Moeng and Rotunno (1990) find that vertical velocity skewness is sensitive to the type of surface boundary conditions and also varies with Reynolds number in direct numerical simulations.

The definition of vertical velocity skewness is
e12
where w is the total velocity. To examine the impact of grid resolution on Sw, the solutions from the different simulations are analyzed with the caveat that we use the resolved or filtered vertical velocity . Hence we compute the resolved skewness
e13
Since typical LES uses Smagorinsky closures with SGS fluxes parameterized at the second moment level, SGS triple moments are unknown and thus there is not a clear definition of “subgrid-scale skewness” in an LES.

Vertical profiles of are shown in Fig. 9. These profiles exhibit a clear and striking dependence on grid resolution; near the surface (z/zi < 0.15) decreases and eventually becomes (unrealistically) negative on the coarse meshes. Meanwhile as z/zi → 1 an opposite trend is observed. With decreasing grid resolution becomes larger and shows a pronounced maximum below the inversion. In the PBL interior (0.1 < z/zi < 0.9), the skewness estimates appear to converge when zi/(CsΔf) > 310 or greater (i.e., when the mesh is greater than or equal to 2563). Near the lower boundary (z/zi < 0.4) the skewness estimates on the 2563, 5123, and 10243 meshes are in good agreement with the few available observations. Above z/zi > 0.75, we have no compelling explanation for the differences between the fine-mesh LES predictions and the few observations but note that the presence of wind shear reduces vertical velocity skewness (Fedorovich et al. 2001, 2004; Lothon et al. 2010). Also, the temporal averaging needed to obtain reliable skewness estimates increases with z (Lenschow et al. 1994), which adds uncertainty to the observations of Sw in the upper PBL. Recently, Lenschow et al. (2011) analyzed vertical velocity collected from a ground-based lidar, over a wider range of shear and convective forcing, and find that their measurements of Sw bracket our LES results.

Fig. 9.
Fig. 9.

Effect of mesh resolution on resolved vertical velocity skewness . The lines legend indicates the mesh size of the various simulations. The skewness is computed using the resolved (or filtered) vertical velocity field . The observations are taken from the results provided in Moeng and Rotunno (1990).

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

Our interpretation of Fig. 9 hinges on the behavior and modeling of the subgrid-scale fluxes in LES. To expose this dependence we introduce the definitions of the third- and second-order SGS moments
e14a
e14b
where indicates a spatially filtered variable. Because the filtering operator commutes with ensemble averaging—that is, —the total skewness given by (12) is next written in terms of resolved and subgrid contributions defined by (14):
e15
Algebraic manipulation of (15) utilizing (14) leads to
e16
where is the resolved-scale skewness (13) and
e17a
e17b
are nondimensional second- and third-order SGS moments. Equation (16) is useful—it defines the total skewness in terms of LES resolved and subgrid-scale variables.

To evaluate the importance of the SGS moments to vertical velocity skewness we filter the 10243 simulation results to produce resolved and SGS variables on a coarser mesh. This step is justified since the LES solutions for vertical velocity are converged at this mesh resolution with a negligible contribution from the SGS (see Fig. 6a). The vertical velocity field from cases E and F are filtered in horizontal x–y planes to a resolution of 642 using a sharp spectral filter—no filtering is applied in the z direction. Tong et al. (1998) show that 2D filtering in a plane is a good approximation to 3D filtering. As an independent check we verified that the filtered fields satisfy (16) exactly.

Vertical profiles of skewness and SGS moments constructed from the filtered 10243 simulation (referred to as case Ff) are presented in Fig. 10; results obtained from filtering case E are similar. The skewness estimates from Ff are similar to the comparable 643 coarse simulation result (i.e., small in the surface layer and large near the inversion) but exhibit important quantitative differences. In the surface layer, the skewness from case Ff is always positive except very near the ground, in contrast to simulation B. This is in agreement with our physical expectation. Also the skewness from Ff matches the high-resolution result in the mid-PBL. The SGS moments in Fig. 10b illustrate the shortcomings of the coarse 643 simulation (case B). In the surface layer the triple moment is very large, contributing more than 50% to , in the mid-PBL , and near the inversion . Also, is always greater than zero. Overall the SGS “contribution” to skewness given by the ratio on the right-hand side of (16) is >4 in the surface layer and ~1 in mid-PBL, and falls to ~0.8 near the inversion on a mesh with 642 resolution in the x–y directions. Hunt et al. (1988) note that Smagorinsky closures are Gaussian SGS models and hence assume . As a consequence, coarse-mesh LES results predict erroneous values of skewness because of their SGS closure schemes. In general, we find that coarse-mesh LES tends to overpredict , underpredict , and thus overpredict compared to fine-resolution simulations as shown in Fig. 11. When Smagorinsky-type closures are used with LES, the resolution ratio zi/(CsΔf) needs to be greater than 630 to obtain mesh-independent estimates of .

Fig. 10.
Fig. 10.

(a) Skewness from the 10243 simulation (solid), the 10243 simulation filtered in horizontal planes to 642 resolution (dotted), and the 643 simulation (dashed). (b) Third- and second-order SGS moments computed from 10243 simulation, showing , the third-order moment (dotted); , the second-order moment (dashed); and the SGS skewness correction [which appears in (16)] (solid).

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

Fig. 11.
Fig. 11.

Comparison of third- and second-order resolved vertical velocity moments from the 10243 simulation (solid), the 10243 simulation filtered in horizontal planes to 642 resolution (dotted), and the 643 simulation (dashed), showing (a) normalized and (b) normalized .

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

The turbulent transport (term T) in (9a) and (9b) depends on the vertical divergence of the third-order moments and , respectively. Vertical profiles of these moments are given in Fig. 12. As might be anticipated based on their content, the moments (γa, γb) are broadly similar to the (w, θ) variances, respectively. Each displays reasonable convergence in the interior of the PBL on the fine meshes. There is a clear mesh dependence in the inversion layer and also near the surface for the γb moment. This is a consequence of the temperature variance mesh dependence. Hence LES that utilize eddy viscosity closures require a very fine mesh to adequately estimate high-order moments in the inversion and wall regions.

Fig. 12.
Fig. 12.

Effect of mesh resolution on resolved third-order moments (left) and (right) .

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

f. Flow visualization

A complete discussion of the impact of mesh resolution on the formation and dynamics of coherent structures and their connection to the statistical moments in the convective PBL is beyond the scope of the present work. Here we briefly illustrate one aspect of large- and small-scale interaction that can occur in high-resolution LES. In Fig. 13, we observe the classic formation of plumes in a convective PBL. Vigorous thermal plumes near the top of the PBL can trace their roots through the middle of the PBL down to the surface layer. Convergence at the common corners of the hexagonal patterns in the surface layer leads to the formation of strong updrafts that evolve into large-scale plumes that fill and dominate the dynamics of the daytime PBL. Near the inversion a descending shell of motion readily develops around each plume.

Fig. 13.
Fig. 13.

Visualization of the vertical velocity field in a convective PBL at different heights from the 10243 simulation: z/zi = (top left) 0.04, (top right) 0.1, (bottom left) 0.5, and (bottom right) 0.9. The gray scale color bar changes between the panels and is in units of m s−1.

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

Closer inspection of the large-scale flow patterns in Fig. 13 also reveals coherent smaller-scale structures. This is demonstrated in Fig. 14 where we track the evolution of 105 particles over about 400 s. Over the limited region where the particles are released the flow is dominated by a persistent line of larger-scale upward convection. On either side of the convection line descending motion develops and near the surface these downdrafts turn laterally and converge. The outcome of this surface layer convergence spawns many small-scale vertically oriented vortices that resemble dust devils. These rapidly rotating vortices are readily observed, persist in time, and rotate in both clockwise and counterclockwise directions. Often the vortices coalesce in a region where a coherent thermal plume erupts. Coarse-mesh LES hints at these coherent vortices but fine-resolution simulations allow a detailed examination of their dynamics within the larger-scale flow. Previously, Kanak (2005) observed the formation of dust devils in convective simulations, but in small computational domains O(750 m).

Fig. 14.
Fig. 14.

Visualization of 105 particles randomly released in a convective PBL at z/zi ~ 0.01 over a limited horizontal extent from the 10243 simulation of convection. The viewed area, ~3.8% of the total horizontal domain, is the topmost left corner from the top-left panel of Fig. 13. Notice the evolution of the larger-scale line of convection into small-scale vortical motions that resemble dust devils. Time advances from left to right beginning along the top row of images. The images are 71.6, 151, and 390 s after the initial release. Vertical vorticity is shown in the bottom-right panel at t = 390 s. The gray scale bar is in units of s−1.

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

5. Summary

A highly parallel large-eddy simulation (LES) code for the atmospheric boundary layer is developed based on a high-Reynolds number Boussinesq flow model with a fully rough lower boundary. The numerical scheme employs pseudospectral differencing in horizontal planes and solves an elliptic pressure Poisson equation utilizing 2D domain decomposition. Despite these global operations, the code exhibits both weak and strong scaling over a wide range of problem sizes with scaling tests are carried out using as many 16 384 processors (see the appendix).

This code is used to carry out a grid sensitivity study of a daytime convective PBL for a wide range of meshes varying from 323 to 10243. Based on the variation of the second-order statistics, spectra, and entrainment statistics we find that the 3D time-dependent LES solutions numerically converge as the mesh is refined for this canonical problem. In the boundary layer interior (0.1 < z/zi < 0.9, where zi is the boundary layer height), the total variances and temperature flux have effectively converged when the mesh resolution is 2563 or greater. The convergence of the total vertical velocity is very good. For our mesh of 2563, the ratio zif > 60 or zi/(CsΔf) > 310, where Δf is the LES filter width and Cs is the Smagorinsky constant. In this regime, the scale separation between the energy containing eddies and the filter cutoff scale is sufficiently wide that the large-eddy Reynolds number Re ~ (zif)4/3 and the parameterized viscous dissipation (e is the subgrid-scale energy) approaches a mesh-independent constant. Two-dimensional spectra of the vertical and horizontal velocities in horizontal planes scale as (kh is the horizontal wavenumber) over almost two decades at the highest resolution. Thus, the LES solutions show clear Kolmogorov inertial subrange scaling, which is the basis of most high-Reynolds number subgrid-scale modeling. Near the rough lower surface and in the entrainment zone, the total (resolved plus subgrid) temperature variance increases with mesh refinement. This is partly a consequence of the subgrid-scale model, which does not employ a prognostic equation for subgrid-scale temperature variance. Potentially, this can be improved by utilizing a fuller set of rate equations for subgrid-scale variables (e.g., Wyngaard 2004b; Hatlee and Wyngaard 2007).

The entrainment rate determined from the time variation of the boundary layer height we = dzi/dt is a sensitive measure of the LES solution convergence. The LES estimates of entrainment velocity become mesh independent when the vertical grid resolution is able to capture both the mean structure of the overlying inversion and the turbulence. The entrainment rate increases with decreasing mesh resolution because of inadequate resolution of the mean temperature gradients in the inversion. For all mesh resolutions used, the vertical temperature flux varies linearly over the boundary layer with the minimum temperature flux ≈ −0.2 of the surface flux. Thus, these scalar-flux properties are not adequate to judge the convergence of LES solutions.

The variation of third-order moments, often used to interpret PBL dynamics, depends on the grid resolution; skewness of resolved vertical velocity highlights the grid resolution sensitivity. The mesh dependence of is a consequence of a Smagorinsky closure that neglects third-order SGS moments of vertical velocity. Interrogation of the 10243 simulations shows the subgrid-scale correction to vertical velocity skewness is greater than one in the surface layer, near unity in mid-PBL, and less than one near the inversion. Simulations with 5123 mesh points or more are needed to estimate vertical velocity skewness and higher-order moments from the resolved LES flow fields. Flow visualization of the 10243 simulations shows the coupling between large-scale thermal plumes and small-scale vortical motions that resemble dust devils. The dust devil cores tend to develop in the branches or spokes of the surface updrafts.

The criterion zi/(CsΔf) > 310 proposed here for simulations of convective boundary layers needs to be tested for simulations of boundary layers dominated by shear, stable stratification, cloudy boundary layers, and boundary layers with surface heterogeneity where the energy containing eddies are concentrated at scales smaller than the boundary layer height zi.

Acknowledgments

We thank Chin-Hoh Moeng, Harm Jonker, and Jeff Weil for their insights and suggestions, which improved the present work. The comments by the anonymous reviewers are appreciated. PPS was partially supported by the Office of Naval Research and by the National Science Foundation through the National Center for Atmospheric Research. EGP acknowledges partial support from the Army Research Office, the National Science Foundation’s Science and Technology Center for Multi-Scale Modeling of Atmospheric Processes, and NCAR’s BEACHON program. This research used resources of the National Energy Research Scientific Computing Center, which is supported by the Office of Science of the U.S. Department of Energy under Contract DE-AC02-05CH11231. Computer time was also provided by NCAR and the Department of Defense.

APPENDIX

Algorithm Parallelization

a. Domain decomposition

The parallelization of the LES algorithm is based on the following criteria: 1) to accomplish 2D domain decomposition using solely the Message Passing Interface (MPI) (Aoyama and Nakano 1999); 2) to preserve pseudospectral differencing in x–y planes using fast Fourier transforms (FFTs); and 3) to maintain a Boussinesq incompressible flow model. The ability to use 2D domain decomposition is a significant advantage in pseudospectral simulation codes as it allows direct numerical simulations of isotropic turbulence on meshes of 20483 or more (Pekurovsky et al. 2006). A sketch of the domain decomposition layout that conforms to our constraints is given in Fig. A1. We mention that 2D domain decomposition in x–y planes is often used with low-order finite-difference schemes (Raasch and Schröter 2001) and mesoscale codes that adopt compressible equations (Michalakes et al. 2005).

Fig. A1.
Fig. A1.

The 2D domain decomposition on nine processors: (a) base state with y–z decomposition, (b) x–z decomposition used for computation of y derivatives and 2D planar FFT, and (c) x–y decomposition used in the tridiagonal matrix inversion of the pressure Poisson equation.

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

In our 2D domain decomposition, each processor operates on constricted three-dimensional “bricks or pencils” subsampled in x, y, or z directions. Brick-to-brick communication is a combination of transposes and ghost point exchange. To preserve pseudospectral differencing in the horizontal directions a custom MPI matrix transpose was designed and implemented. Other nonlocal schemes—such as compact finite difference (Lele 1992) or fully spectral direct numerical simulation codes (Werne and Fritts 1999)—require similar communication patterns. Given a field f(x, y, z) discretized at (Nx, Ny, Nz) locations, our transpose routines perform the forward and inverse operations
ea1
using a subset of horizontal processors as shown in Figs. A1a and A1b. In (A1) and the following equations, subscripts (·)s,e denote starting and ending locations in the (x, y, z) directions. The data transpose shown schematically in Figs. A1a and A1b only requires local communication, that is, communication between processors in groups [0, 1, 2], [3, 4, 5], and [6, 7, 8]. Derivatives ∂f/∂y, which are needed in physical space, are computed using this sequence of steps:
  1. forward x to y transpose ,
  2. FFT derivative ∂fT/∂y, and
  3. inverse y to x transpose ∂fT/∂y → ∂f/∂y.
Existing serial 1D FFT routines for real and complex arrays are used as in previous implementations (see http://www.cisl.ucar.edu/css/software/fftpack5/). With this algorithm, so-called ghost points used in computing derivatives ∂f/∂z are only needed on the top and bottom faces of each brick in Fig. A1a.
The 2D decomposition of the computational domain also impacts the pressure Poisson equation solver. In an incompressible Boussinesq fluid model the pressure is a solution of the elliptic equation (see section 2)
ea2
where the source term r is the numerical (discrete) divergence of the unsteady momentum equations (e.g., Sullivan et al. 1996). The solution for begins with a standard forward 2D Fourier transform of (A2):
ea3
where the 2D Fourier transform for or r is
ea4
and (kx, ky) are horizontal wavenumbers. At this stage the data layout on each processor is as shown in Fig. A1b. Next, custom routines carry out forward ky to z and inverse z to ky matrix transposes on the source term of (A3):
ea5
Again notice the communication pattern needed to transpose from Fig. A1b to A1c is accomplished locally by processors in groups [0, 3, 6], [1, 4, 7], and [2, 5, 8]. The continuous storage of along the z direction allows standard tridiagonal matrix inversion for pairs of horizontal wavenumbers on each processor. This step is repeated for all pairs of horizontal wavenumbers and provides the transposed field . To recover the pressure field in physical space we retrace our steps: followed by an inverse 2D Fourier transform . In designing the present algorithm, we also considered using the parallel tridiagonal solver described by Gibbs (2004) for the solution of the Poisson equation but found it not as flexible as the present scheme.

With these enhancements our new algorithm allows a very large number of processors O(104) or more to be utilized. No global communication between processors is required; that is, we do not call MPI’s ALL_TO_ALL routine. Instead, the MPI routine SENDRECV is wrapped with FORTRAN statements to accomplish the desired communication pattern. The scheme outlined above introduces more communication but the send–receive messages are smaller and hence large numbers of grid points can be used. Also, the total number of processors is not limited by the number of vertical grid points. This flexibility allows simulations in boxes with large horizontal and small vertical extents. The transpose routines are general and allow arbitrary numbers of mesh points, although the best performance is of course realized when the load is balanced across processors.

b. Scaling

The performance of the code for varying workload as a function of the total number of processors NP is provided in Figs. A2 and A3 for three different machine architectures (NP = NPz × NPxy where NPz and NPxy are the number of processors in the vertical and horizontal directions, respectively). In each figure, the vertical axis is total computational time t × NP divided by total work. Also, Nz is the number of vertical levels and Mx,y is proportional to the FFT work (i.e., Mx,y = Nx,ylogNx,y, with Nx,y being the number of grid points in the x and y directions). Ideal scaling corresponds to a flat line with increasing number of processors. The timing tests illustrate the present scheme exhibits both strong scaling (i.e., where the problem size is held fixed and the number of processors is increased) and weak scaling (i.e., where the problem size grows as the number of processors increases so the amount of work per processor is held constant) over a wide range of problem sizes and is able to use as many as 16 384 processors (i.e., the maximum number available to our application). Further, the results are robust for varying combinations of (NPz, NPxy). Generally, the performance only begins to degrade when the number of processors exceeds about 8 times the minimum of (Nx, Ny, Nz) because of increases in communication overhead.

Fig. A2.
Fig. A2.

Computational time per grid point for different combinations of problem size and 2D domain decomposition for the Cray XT4 (an example of strong scaling), showing problem sizes (a) 5123 (⋄), (b) 10243 (○), (c) 20483 (□), and (d) 30723 (Δ). For a given number of total processors NP the symbols are varying vertical and horizontal decompositions [i.e., different combinations (NPz, NPxy)].

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

Fig. A3.
Fig. A3.

Computational time per grid point for a fixed amount of work per processor (an example of weak scaling). Shown are 60 000 points per processor for the Cray XT4 (○), dual core IBM SP5+ (⋄), and single core IBM SP5 (□), and 524 288 points per processor for the Cray XT4 (Δ). For a fixed number of total processors NP multiple symbols are different combinations of (NPz, NPxy).

Citation: Journal of the Atmospheric Sciences 68, 10; 10.1175/JAS-D-10-05010.1

REFERENCES

  • Andren, A., , A. R. Brown, , P. J. Mason, , J. Graf, , U. Schumann, , C.-H. Moeng, , and F. T. M. Nieuwstadt, 1994: Large-eddy simulation of a neutrally stratified boundary layer: A comparison of four computer codes. Quart. J. Roy. Meteor. Soc., 120, 14571484.

    • Search Google Scholar
    • Export Citation
  • Aoyama, Y., , and J. Nakano, 1999: RS/6000 SP: Practical MPI programming. Tech. Rep. IBM Redbook SG24-5380-00, International Business Machines, 221 pp.

    • Search Google Scholar
    • Export Citation
  • Beare, R. J., and Coauthors, 2006: An intercomparison of large-eddy simulations of the stable boundary layer. Bound.-Layer Meteor., 118, 242272.

    • Search Google Scholar
    • Export Citation
  • Beets, C., , and B. Koren, 1996: Large-eddy simulation with accurate implicit subgrid-scale diffusion. Department of Numerical Mathematics Rep. NM-R9601, Utrecht University, 24 pp.

    • Search Google Scholar
    • Export Citation
  • Betts, A. K., 1974: Reply to comment on the paper ‘Non-precipitating cumulus convection and its parameterization.’ Quart. J. Roy. Meteor. Soc., 100, 469471.

    • Search Google Scholar
    • Export Citation
  • Brasseur, J. G., , and T. Wei, 2010: Designing large eddy simulation of the turbulent boundary layer to capture law-of-the-wall scaling. Phys. Fluids,22, 021303, doi:10.1063/1.3319073.

    • Search Google Scholar
    • Export Citation
  • Bretherton, C. S., and Coauthors, 1999: An intercomparison of radiatively driven entrainment and turbulence in a smoke cloud, as simulated by different numerical models. Quart. J. Roy. Meteor. Soc., 554, 391423.

    • Search Google Scholar
    • Export Citation
  • Bryan, G. H., , J. C. Wyngaard, , and J. M. Fritsch, 2003: Resolution requirements for the simulation of deep moist convection. Mon. Wea. Rev., 131, 23942416.

    • Search Google Scholar
    • Export Citation
  • Celik, I., , M. Klein, , M. Freitag, , and J. Janicka, 2006: Assessment measures for URANS/DES/LES: An overview with applications. J. Turbul., 7, 127.

    • Search Google Scholar
    • Export Citation
  • Chow, F. K., , and P. Moin, 2003: A further study of numerical errors in large-eddy simulations. J. Comput. Phys., 184, 366380.

  • Davis, K. J., , N. Gamage, , C. R. Hagelberg, , D. H. L. C. Kiemle, , and P. P. Sullivan, 2000: An objective method for deriving atmospheric structure from airborne lidar observations. J. Atmos. Oceanic Technol., 17, 14551468.

    • Search Google Scholar
    • Export Citation
  • Deardorff, J. W., 1972a: Numerical investigation of neutral and unstable planetary boundary layers. J. Atmos. Sci., 29, 91115.

  • Deardorff, J. W., 1972b: Three-dimensional numerical modeling of the planetary boundary layer. Workshop on Micrometeorology, D. A. Haugen, Ed., Amer. Meteor. Soc., 271–311.

    • Search Google Scholar
    • Export Citation
  • Deardorff, J. W., 1980: Stratocumulus-capped mixed layers derived from a three-dimensional model. Bound.-Layer Meteor., 18, 495527.

  • de Roode, S. R., , P. G. Duynkerke, , and H. J. J. Jonker, 2004: Large eddy simulation: How large is large enough? J. Atmos. Sci., 61, 403421.

    • Search Google Scholar
    • Export Citation
  • Fedorovich, E., , F. T. M. Nieuwstadt, , and R. Kaiser, 2001: Numerical and laboratory study of a horizontally evolving convective boundary layer. Part I: Transition regimes and development of the mixed layer. J. Atmos. Sci., 58, 7086.

    • Search Google Scholar
    • Export Citation
  • Fedorovich, E., and Coauthors, 2004: Entrainment into sheared convective boundary layers as predicted by different large eddy simulation codes. Preprints, 16th Symp. on Boundary Layer and Turbulence, Portland, ME, Amer. Meteor. Soc., P4.7.

    • Search Google Scholar
    • Export Citation
  • Geurts, B. J., 2001: Modern Simulation Strategies for Turbulent Flow. R. T. Edwards, 327 pp.

  • Geurts, B. J., , and J. Fröhlich, 2002: A framework for predicting accuracy limitations in large-eddy simulation. Phys. Fluids, 14, L41L44.

    • Search Google Scholar
    • Export Citation
  • Gibbs, W. R., 2004: A parallel/recursive algorithm. J. Comput. Phys., 201, 573585.

  • Hatlee, S. C., , and J. C. Wyngaard, 2007: Improved subfilter-scale models from the HATS field data. J. Atmos. Sci., 64, 16941705.

  • Hunt, J. C. R., , J. C. Kaimal, , and J. E. Gaynor, 1988: Eddy structure in the convective boundary layer—New measurements and new concepts. Quart. J. Roy. Meteor. Soc., 482, 827858.

    • Search Google Scholar
    • Export Citation
  • Jonker, H. J. J., , P. G. Duynkerke, , and J. W. M. Cuijpers, 1999: Mesoscale fluctuations in scalars generated by boundary layer convection. J. Atmos. Sci., 56, 801808.

    • Search Google Scholar
    • Export Citation
  • Kaltenbach, H.-J., 1997: Cell aspect ratio dependence of anisotropy measures for resolved and subgrid scale stresses. J. Comput. Phys., 136, 399410.

    • Search Google Scholar
    • Export Citation
  • Kanak, K. M., 2005: Numerical simulation of dust devil–scale vortices. Quart. J. Roy. Meteor. Soc., 131, 12711292.

  • Klemp, J., , and D. Durran, 1983: An upper boundary condition permitting internal gravity wave radiation in numerical mesoscale models. Mon. Wea. Rev., 111, 430444.

    • Search Google Scholar
    • Export Citation
  • Koren, B., 1993: A robust upwind discretization method for advection, diffusion and source terms. Notes on Numerical Fluid Mechanics, Vol. 45, C. B. Vreugdenhil and B. Koren, Eds., Vieweg-Braunschweig, 117–138.

    • Search Google Scholar
    • Export Citation
  • Lele, S. K., 1992: Compact finite difference schemes with spectral-like resolution. J. Comput. Phys., 103, 1642.

  • Lenschow, D. H., , J. C. Wyngaard, , and W. T. Pennell, 1980: Mean-field and second-moment budgets in a baroclinic, convective boundary layer. J. Atmos. Sci., 37, 13131326.

    • Search Google Scholar
    • Export Citation
  • Lenschow, D. H., , J. Mann, , and L. Kristensen, 1994: How long is long enough when measuring fluxes and other turbulence statistics? J. Atmos. Oceanic Technol., 11, 661673.

    • Search Google Scholar
    • Export Citation
  • Lenschow, D. H., , M. Lothon, , S. D. Mayor, , P. P. Sullivan, , and G. Canut, 2011: A comparison of higher-order vertical velocity moments in the convective boundary layer from lidar with in situ measurements and LES. Bound.-Layer Meteor., doi:10.1007/s10546-011-9615-3, in press.

    • Search Google Scholar
    • Export Citation
  • Lilly, D. K., 1967: The representation of small-scale turbulence in numerical simulation experiments. Proc. IBM Scientific Computing Symp. on Environmental Sciences, Yorktown Heights, NY, International Business Machines, 195–210.

    • Search Google Scholar
    • Export Citation
  • Lothon, M., , D. H. Lenschow, , and S. D. Mayor, 2009: Doppler lidar measurements of vertical velocity spectra in the convective planetary boundary layer. Bound.-Layer Meteor., 132, 205226.

    • Search Google Scholar
    • Export Citation
  • Lothon, M., , D. H. Lenschow, , G. Canut, , S. D. Mayor, , and P. P. Sullivan, 2010: Measurements of higher-order turbulence statistics in the daytime convective boundary layer derived from a ground-based Doppler lidar. Proc. Int. Symp. for the Advancement of Boundary Layer Remote Sensing, Paris, France, ISARS. [Available online at http://www.isars2010.uvsq.fr/images/stories/PosterExtAbstracts/P_TUR03_Lothon.pdf.]

    • Search Google Scholar
    • Export Citation
  • Lundquist, K. A., , F. K. Chow, , and J. K. Lundquist, 2010: An immersed boundary method for the Weather Research and Forecasting model. Mon. Wea. Rev., 138, 796817.

    • Search Google Scholar
    • Export Citation
  • Mason, P. J., , and A. R. Brown, 1999: On subgrid models and filter operations in large-eddy simulations. J. Atmos. Sci., 56, 21012114.

    • Search Google Scholar
    • Export Citation
  • Meneveau, C., , and J. Katz, 2000: Scale-invariance and turbulence models for large-eddy simulations. Annu. Rev. Fluid Mech., 32, 132.

  • Meyers, J., , B. J. Geurts, , and P. Sagaut, 2007: A computational error-assessment of central finite-volume discretizations in large-eddy simulation using a Smagorinsky model. J. Comput. Phys., 227, 156173.

    • Search Google Scholar
    • Export Citation
  • Michalakes, J., , J. Dudhia, , D. Gill, , T. Henderson, , J. Klemp, , W. Skamarock, , and W. Wang, 2005: The Weather Research and Forecast Model: Software architecture and performance. Proceedings of the Eleventh ECMWF Workshop on the Use of High Performance Computing in Meteorology, W. Zwieflhofer and G. Mozdzynski, Eds., World Scientific, 156–168.

    • Search Google Scholar
    • Export Citation
  • Mironov, D. V., 2009: Turbulence in the lower troposphere: Second-order closure and mass-flux modelling frameworks. Interdisciplinary Aspects of Turbulence, W. Hillebrandt and F. Kupka, Eds., Lecture Notes in Physics, Vol. 756, Springer-Verlag, 161–221.

    • Search Google Scholar
    • Export Citation
  • Mironov, D. V., , V. M. Gryanik, , C.-H. Moeng, , D. J. Olbers, , and T. H. Warncke, 2000: Vertical turbulence structure and second-moment budgets in convection with rotation: A large-eddy simulation study. Quart. J. Roy. Meteor. Soc., 126, 477515.

    • Search Google Scholar
    • Export Citation
  • Moeng, C.-H., 1984: A large-eddy-simulation model for the study of planetary boundary-layer turbulence. J. Atmos. Sci., 41, 20522062.

    • Search Google Scholar
    • Export Citation
  • Moeng, C.-H., , and J. C. Wyngaard, 1988: Spectral analysis of large-eddy simulations of the convective boundary layer. J. Atmos. Sci., 45, 35733587.

    • Search Google Scholar
    • Export Citation
  • Moeng, C.-H., , and R. Rotunno, 1990: Vertical velocity skewness in the buoyancy-driven boundary layer. J. Atmos. Sci., 47, 11491162.

  • Moeng, C.-H., , and P. P. Sullivan, 1994: A comparison of shear- and buoyancy-driven planetary boundary layer flows. J. Atmos. Sci., 51, 9991022.

    • Search Google Scholar
    • Export Citation
  • Moeng, C.-H., , and P. P. Sullivan, 2002: Large-eddy simulation. Encyclopedia of Atmospheric Sciences, J. R. Holton, J. Pyle, and J. A. Curry, Eds., Academic Press, 1140–1150.

    • Search Google Scholar
    • Export Citation
  • Moeng, C.-H., , M. A. LeMone, , M. F. Khairoutdinov, , S. K. Krueger, , P. A. Bogenschutz, , and D. A. Randall, 2009: The tropical marine boundary layer under a deep convection system: A large-eddy simulation study. J. Adv. Model. Earth Syst., 1 (16), doi:10.3894/JAMES.2009.1.16.

    • Search Google Scholar
    • Export Citation
  • Muschinski, A., 1996: A similarity theory of locally homogenous and isotropic turbulence generated by a Smagorinsky-type LES. J. Fluid Mech., 325, 239260.

    • Search Google Scholar
    • Export Citation
  • National Science Foundation, 2007: Cyberinfrastructure vision for 21st century discovery. Tech. Rep. NSF 07–28, NSF Cyberinfrastructure Council. [Available online at http://www.nsf.gov/pubs/2007/nsf0728/index.jsp.]

    • Search Google Scholar
    • Export Citation
  • Nieuwstadt, F. T. M., , P. J. Mason, , C. H. Moeng, , and U. Schumann, 1993: Large-eddy simulation of the convective boundary layer: A comparison of four computer codes. Turbulent Shear Flows 8, F. Durst, Ed., Springer-Verlag, 343–367.

    • Search Google Scholar
    • Export Citation
  • Patton, E. G., , P. P. Sullivan, , and C.-H. Moeng, 2005: The influence of idealized heterogeneity on wet and dry planetary boundary layers coupled to the land surface. J. Atmos. Sci., 62, 20782097.

    • Search Google Scholar
    • Export Citation
  • Pekurovsky, D., , P. K. Yeung, , D. Donzis, , W. Pfeiffer, , and G. Chukkapallli, 2006: Scalability of a pseudospectral DNS turbulence code with 2D domain decomposition on Power4+/Federation and Blue Gene systems. ScicomP12 and SP-XXL, Boulder, CO, International Business Machines. [Available online at http://www.spscicomp.org/ScicomP12/Presentations/User/Pekurovsky.pdf.]

    • Search Google Scholar
    • Export Citation
  • Pope, S. B., 2000: Turbulent Flows. Cambridge University Press, 771 pp.

  • Raasch, S., , and M. Schröter, 2001: PALM—A large-eddy simulation model performing on massively parallel computers. Meteor. Z., 10, 363372.

    • Search Google Scholar
    • Export Citation
  • Schmidt, H., , and U. Schumann, 1989: Coherent structure of the convective boundary layer derived from large-eddy simulations. J. Fluid Mech., 200, 511562.

    • Search Google Scholar
    • Export Citation
  • Scotti, A., , C. Meneveau, , and D. K. Lilly, 1993: Generalized Smagorinsky model for anisotropic grids. Phys. Fluids A, 5, 23062308.

  • Silva Lopes, A., , and J. M. L. M. Palma, 2002: Numerical simulation of isotropic turbulence using a collocated approach and a nonorthogonal grid system. J. Comput. Phys., 175, 713738.

    • Search Google Scholar
    • Export Citation
  • Spalart, P. R., , R. D. Moser, , and M. M. Rogers, 1991: Spectral methods for the Navier–Stokes equations with one infinite and two periodic directions. J. Comput. Phys., 96, 297324.

    • Search Google Scholar
    • Export Citation
  • Sreenivasan, K. R., , A. Bershadskii, , and J. J. Niemela, 2002: Mean wind and its reversal in thermal convection. Phys. Rev. E, 65, 056306, doi:10.1103/PhysRevE.65.056306.

    • Search Google Scholar
    • Export Citation
  • Stevens, B., and Coauthors, 2005: Evaluation of large-eddy simulations via observations of nocturnal marine stratocumulus. Mon. Wea. Rev., 133, 14431462.

    • Search Google Scholar
    • Export Citation
  • Sullivan, P. P., , and J. C. McWilliams, 2010: Dynamics of winds and currents coupled to surface waves. Annu. Rev. Fluid Mech., 42, 1942.

    • Search Google Scholar
    • Export Citation
  • Sullivan, P. P., , J. C. McWilliams, , and C.-H. Moeng, 1994: A subgrid-scale model for large-eddy simulation of planetary boundary-layer flows. Bound.-Layer Meteor., 71, 247276.

    • Search Google Scholar
    • Export Citation
  • Sullivan, P. P., , J. C. McWilliams, , and C.-H. Moeng, 1996: A grid nesting method for large-eddy simulation of planetary boundary layer flows. Bound.-Layer Meteor., 80, 167202.

    • Search Google Scholar
    • Export Citation
  • Sullivan, P. P., , C.-H. Moeng, , B. Stevens, , D. H. Lenschow, , and S. D. Mayor, 1998: Structure of the entrainment zone capping the convective atmospheric boundary layer. J. Atmos. Sci., 55, 30423064.

    • Search Google Scholar
    • Export Citation
  • Sullivan, P. P., , T. W. Horst, , D. H. Lenschow, , C.-H. Moeng, , and J. C. Weil, 2003: Structure of subfilter-scale fluxes in the atmospheric surface layer with application to large-eddy simulation modeling. J. Fluid Mech., 482, 101139.

    • Search Google Scholar
    • Export Citation
  • Sullivan, P. P., , J. C. McWilliams, , and W. K. Melville, 2007: Surface gravity wave effects in the oceanic boundary layer: Large-eddy simulation with vortex force and stochastic breakers. J. Fluid Mech., 593, 405452.

    • Search Google Scholar
    • Export Citation
  • Sullivan, P. P., , J. B. Edson, , T. Hristov, , and J. C. McWilliams, 2008: Large-eddy simulations and observations of atmospheric marine boundary layers above non-equilibrium surface waves. J. Atmos. Sci., 65, 12251245.

    • Search Google Scholar
    • Export Citation
  • Tong, C., , J. C. Wyngaard, , S. Khanna, , and J. G. Brasseur, 1998: Resolvable- and subgrid-scale measurement in the atmospheric surface layer: Technique and issues. J. Atmos. Sci., 55, 31143126.

    • Search Google Scholar
    • Export Citation
  • Townsend, A. A., 1976: The Structure of Turbulent Shear Flow. Cambridge University Press, 429 pp.

  • Weil, J. C., 1988: Dispersion in the convective boundary layer. Lectures on Air Pollution Modeling, A. Venkatram and J. Wyngaard, Eds., Amer. Meteor. Soc., 167–227.

    • Search Google Scholar
    • Export Citation
  • Weil, J. C., 1990: A diagnosis of the asymmetry in top-down and bottom-up diffusion in a Lagrangian stochastic model. J. Atmos. Sci., 47, 501515.

    • Search Google Scholar
    • Export Citation
  • Werne, J., , and D. C. Fritts, 1999: Stratified shear turbulence: Evolution and statistics. Geophys. Res. Lett., 26, 439442.

  • Wyngaard, J. C., 1998: Boundary-layer modeling: History, philosophy, and sociology. Clear and Cloudy Boundary Layers, A. A. M. Holtslag and P. G. Duynkerke, Eds., Royal Netherlands Academy of Arts and Sciences, 325–332.

    • Search Google Scholar
    • Export Citation
  • Wyngaard, J. C., 2004a: Changing the face of small-scale meteorology. Atmospheric Turbulence and Mesoscale Meteorology, E. Federovich, R. Rotunno, and B. Stevens, Eds., Cambridge University Press, 17–34.

    • Search Google Scholar
    • Export Citation
  • Wyngaard, J. C., 2004b: Toward numerical modeling in the “terra incognita.” J. Atmos. Sci., 61, 18161826.

  • Wyngaard, J. C., 2010: Turbulence in the Atmosphere. Cambridge University Press, 393 pp.

Save