1. Introduction
State-of-the-art atmosphere models prognose an increasing number of tracers, which involves solving the continuity equation of each tracer. Tracers in atmosphere models typically include different forms of water (humidity, cloud water, cloud ice, rain, snow, etc.). Models with a comprehensive treatment of aerosols explicitly solve a number of aerosol continuity equations, and models with comprehensive chemistry prognose many chemical species. For example, NCAR’s Community Atmosphere Model (CAM), version 5 (Neale et al. 2012), in its standard configuration solves 30 continuity equations in addition to the continuity equation for air and over 120 continuity equations when run with comprehensive chemistry. Hence, the number of continuity equations in state-of-the-art models easily outnumbers the remaining equations of motion [momentum equations and thermodynamic equation(s)]. Consequently, the computational cost of tracer transport may dominate the overall cost of the model even if tracer transport is supercycled (i.e., run with a larger time step than the dynamics).
Many tracer distributions in the atmosphere have complex spatial structures associated with them. For example, photolysis-driven chemical species exhibit large gradients near the solar terminator as they are destroyed or created by sunlight and moist processes; for example, convection can introduce grid-scale structure in the water variables. It is usually challenging for transport algorithms to accurately represent the transport process for tracer distributions with large gradients and grid-scale structure. It is important that the transport scheme is not a spurious source or sink of mass. For coupling with parameterizations, it is desirable that the transport scheme provides physically realizable solutions; in particular, it should not produce negative values for a positive definite tracer. To avoid overshoots near large gradients, it is desirable that the scheme is monotone (also loosely referred to as shape preserving or nonoscillatory). For the representation of aerosols and chemical species, it is important that the transport scheme does not spuriously perturb relations between tracers (if they are functionally related; e.g., see Lin and Rood 1996; Lauritzen and Thuburn 2012). For a more complete discussion on desirable properties for transport schemes intended for climate and climate/chemistry applications, see, for example, Lauritzen et al. (2011).
To assess the transport scheme’s ability to advect rough distributions and maintain nonlinear correlations between trace species, Lauritzen et al. (2012) created an idealized test case suite for modelers to assess desirable properties for their transport algorithm in an idealized setup. This test case suite used an idealized 2D wind field from Nair and Lauritzen (2010), with a range of initial condition distributions for tracers ranging from smooth to distributions with discontinuous gradients and a wide range of existing and new diagnostics. Results from over a dozen transport algorithms are presented in Lauritzen et al. (2014b).
To improve the scalability of CAM based on the finite-volume dynamical core (CAM-FV; Lin 2004), the spectral element (SE) dynamical core was imported into CAM from High-Order Methods Modeling Environment (HOMME; Dennis et al. 2005; Nair et al. 2009; Taylor and Fournier 2010) and is referred to as CAM-SE. CAM-SE is based on a continuous Galerkin finite-element (or spectral element) method (Taylor et al. 1997) and is discretized using a cubed-sphere tiling of the sphere. Petascale scalability of CAM-SE was demonstrated in Dennis et al. (2012). Also, CAM-SE provided refined mesh functionality in CAM (e.g., St-Cyr et al. 2008; Zarzycki et al. 2014), improved accuracy of idealized baroclinic wave simulations (Lauritzen et al. 2010a), conserved total energy to time-truncation errors (Taylor 2011), and improved global axial angular momentum budgets compared to CAM-FV (Lauritzen et al. 2014a). That said, the tracer transport component of CAM-SE appeared to be less accurate than CAM-FV for nonsmooth tracer distributions (Lauritzen et al. 2014b; Hall et al. 2016). In an idealized terminator test in which two species react according to a reaction coefficient proportional to the solar terminator while being transported by an idealized 2D flow, CAM-SE produces errors larger than CAM-FV (Lauritzen et al. 2015). The errors are due to the limiter used for tracer transport that prevents oscillatory (and, in particular, negative) solutions for tracers (Guba et al. 2014). In terms of computational throughput, it was also found that CAM-SE was slower for large tracer counts (at least when run with core counts similar to the nonscalable CAM-FV).
It is the overall purpose of this paper to improve the accuracy and efficiency of tracer transport in CAM-SE. To improve the accuracy and efficiency of multitracer transport, the Conservative Semi-Lagrangian Multitracer (CSLAM; Lauritzen et al. 2010b) transport scheme was implemented in HOMME (Erath et al. 2012, 2013). Later, an alternative inherently conservative semi-Lagrangian scheme called spectral element Lagrangian transport (SPELT; Erath and Nair 2014) was implemented in HOMME, which easily supports mesh-refinement applications and has a simpler search algorithm compared to the original CSLAM (note that the new version of CSLAM presented here does not have a comprehensive search algorithm). Both SPELT and CSLAM are fully 2D finite-volume semi-Lagrangian schemes. Semi-Lagrangian schemes based on dimensional splitting along coordinate lines (Putman and Lin 2007) or Lagrangian translations of coordinate lines (conservative cascade methods; e.g., Nair et al. 2002; Zerroukat et al. 2002; Shashkin and Tolstykh 2016) are alternatives to fully 2D methods that are more easily extensible to 3D methods. Cascade schemes are computationally efficient and designed for structured orthogonal grid systems. It is unclear how to extend the cascade methods to nonorthogonal grids—in particular, the gnomonic cubed-sphere grid used here—as the grid lines are discontinuous near the cubed-sphere edges and corners. All of these schemes are stable for long time steps, inherently mass conservative, and shape preserving; some of them preserve linear relations between tracers even with a shape-preserving limiter (e.g., CSLAM) and are efficient for large tracer counts as the upstream Lagrangian grids can be reused for each additional tracer. Specific to this study, we note that in 2D idealized test cases, the CSLAM scheme is more accurate than CAM-SE in terms of the diagnostics used in the test case suite presented in Lauritzen et al. (2012).
For 3D applications in CAM-SE, the winds for CSLAM are provided by CAM-SE, which creates an inconsistency between the continuity equation for air (solved with the SE method) and tracers (solved with CSLAM). In other words, if a tracer is uniformly one, then the solution for tracer mass, computed with CSLAM, does not reduce to the CAM-SE continuity equation for air. Inconsistency can lead to loss of tracer mass conservation and/or unphysical solutions (nonmonotone). This problem of the discretization of the transport equation not being consistent with the airmass equation has been known for some time (Rood 1987; Lin and Rood 1996; Gross et al. 2002). Discretizations that are consistent are also referred to in the literature as free-stream preserving, constancy preserving, or compatible. Solutions have been proposed for shallow-water applications in which correction terms in the velocity fields have been introduced to enforce consistency (e.g., Deleersnijder and Lebon 2001) or using a corrective mass flux in the transport scheme (e.g., Dawson 1999).
In this paper, we present a CSLAM-based algorithm in which CSLAM is made consistent with the SE continuity equation for air so that the CSLAM solution to the tracer continuity equations is mass conservative, monotone, and consistent. This is achieved by diagnosing CAM-SE airmass fluxes through CSLAM control volume edges and using that information to construct CSLAM control volumes for which the airmass flux implied by CSLAM matches CAM-SE fluxes to round off. Consistency is effectively achieved by finding an upstream departure grid for which the CSLAM solution for air mass matches the CAM-SE solution for air mass to round off. Rather than manipulating velocity components or adjusting fluxes a posteriori, the consistency is enforced in Lagrangian space. Note that the algorithm for enforcing consistency in CSLAM is general. Any airmass fluxes (so not necessarily CAM-SE diagnosed fluxes) that satisfy the Lipschitz criterion (also known as the deformational Courant number by which the upstream trajectories do not cross intersect in a single time step; Pudykiewicz et al. 1985; Kuo and Williams 1990) and for which the Courant number is less than one can be used.
The paper is organized as follows: The continuity equation for CSLAM and SE are introduced in section 2 as well as the consistent coupling problem formulation. In section 3, the consistent SE-CSLAM algorithm is presented, and in section 4, we present results from idealized baroclinic wave simulations with inert and reactive tracers in order to assess accuracy and computational efficiency. A summary is provided in section 5.
2. Methods
a. Finite-volume continuity equation
Assume a floating Lagrangian vertical coordinate that initially coincides with the hybrid-sigma coordinates (Lin 2004; Starr 1945). The air mass in each layer is 
A graphical illustration of (a) the finite-volume Lagrangian discretization of the continuity equation and (b)–(e) the Eulerian finite-volume flux-form discretization. In (a), the upstream Lagrangian area 
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
A graphical illustration of (a) the finite-volume Lagrangian discretization of the continuity equation and (b)–(e) the Eulerian finite-volume flux-form discretization. In (a), the upstream Lagrangian area
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
A graphical illustration of (a) the finite-volume Lagrangian discretization of the continuity equation and (b)–(e) the Eulerian finite-volume flux-form discretization. In (a), the upstream Lagrangian area
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
In the CSLAM discretization, the vertices of 
b. SE continuity equation
(a) The 
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
(a) The
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
(a) The
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
c. Problem formulation
The challenge is to couple the CSLAM continuity equation for tracers with the SE continuity equation for air without violating the following:
Tracer mass conservation
Consistency (aka compatibility, free-stream preservation, constancy preservation)
Tracer shape preservation
Linear correlations between tracers (e.g., Lin and Rood 1996; Lauritzen and Thuburn 2012)
If one can find an upstream Lagrangian grid 
The problem of developing swept areas to match the SE fluxes may initially seem intractable, as the two methods appear very different (SE is often thought of as a pointwise method in contrast with the volumetric approach of finite-volume methods). However, local construction of the swept areas is made possible by the fact that, on the level of individual spectral elements, there is a well-defined notion of fluxes between adjacent elements. It is exactly this correspondence that allows SE methods to maintain global mass conservation on a local level.
3. Consistent CSLAM flux algorithm
The algorithm has been designed to avoid (as much as possible) conditional statements in the code for control volumes at the cubed-sphere edges. Hence, there is a substantial amount of notation that needs to be introduced. The basic algorithm is explained in Fig. 3.
Step-by-step example of the consistent SE-CSLAM algorithm for face 3. (a),(b) Compute the CSLAM flux perpendicular to each face by iteration. (c) The intersection between the upstream perpendicular flux faces defines the departure points for CSLAM (dark blue filled circles). (d) A degree of freedom for the swept flux area is introduced at the intersection between the perpendicular swept y-flux area and the coordinate line (red circle filled with yellow). (e),(f) By iteration, the degree of freedom is moved up and down along the coordinate axis until the flux area exactly matches the SE flux.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
Step-by-step example of the consistent SE-CSLAM algorithm for face 3. (a),(b) Compute the CSLAM flux perpendicular to each face by iteration. (c) The intersection between the upstream perpendicular flux faces defines the departure points for CSLAM (dark blue filled circles). (d) A degree of freedom for the swept flux area is introduced at the intersection between the perpendicular swept y-flux area and the coordinate line (red circle filled with yellow). (e),(f) By iteration, the degree of freedom is moved up and down along the coordinate axis until the flux area exactly matches the SE flux.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
Step-by-step example of the consistent SE-CSLAM algorithm for face 3. (a),(b) Compute the CSLAM flux perpendicular to each face by iteration. (c) The intersection between the upstream perpendicular flux faces defines the departure points for CSLAM (dark blue filled circles). (d) A degree of freedom for the swept flux area is introduced at the intersection between the perpendicular swept y-flux area and the coordinate line (red circle filled with yellow). (e),(f) By iteration, the degree of freedom is moved up and down along the coordinate axis until the flux area exactly matches the SE flux.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
a. Departure point algorithm
The swept area 
Set
Set
- Compute
Set
If
Done: set
Perpendicular swept flux situations in which the displacement γ for the center vertex (
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
Perpendicular swept flux situations in which the displacement γ for the center vertex (
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
Perpendicular swept flux situations in which the displacement γ for the center vertex (
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
b. Final swept flux area
The algorithm for calculating the final swept areas is the same for each side with appropriate index change. Hence, we describe the algorithm only for side 1, 
(a) Definition of displacement variable d for side 1 in cell 
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
(a) Definition of displacement variable d for side 1 in cell
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
(a) Definition of displacement variable d for side 1 in cell
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
If the departure point
- If the departure point
- (green filled circle), where
- and
- (green filled circle), where
(cyan filled circle).
- If both departure points,
- There is no crossing between the upstream flux side and the coordinate lines. We introduce a point halfway between the departure points:
and it is perturbed along a vector perpendicular to the flux side—that is, a 90° counterclockwise rotation of the vector
A depiction of the categorization of possible flow cases defined in (15). The light blue shaded area is the first-guess swept flux. The dark blue filled circles are the departure points (that remain fixed) and the yellow filled circles are moved along the coordinate lines (according to red arrows) until the swept airmass flux matches the SE mass flux to round off. The middle and right columns showcase where one or two departure points coincide with a grid line.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
A depiction of the categorization of possible flow cases defined in (15). The light blue shaded area is the first-guess swept flux. The dark blue filled circles are the departure points (that remain fixed) and the yellow filled circles are moved along the coordinate lines (according to red arrows) until the swept airmass flux matches the SE mass flux to round off. The middle and right columns showcase where one or two departure points coincide with a grid line.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
A depiction of the categorization of possible flow cases defined in (15). The light blue shaded area is the first-guess swept flux. The dark blue filled circles are the departure points (that remain fixed) and the yellow filled circles are moved along the coordinate lines (according to red arrows) until the swept airmass flux matches the SE mass flux to round off. The middle and right columns showcase where one or two departure points coincide with a grid line.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
The above algorithm constitutes a local iteration problem since a point along the side of the swept area is perturbed to obtain CSLAM fluxes consistent with the diagnosed SE fluxes. In other words, the vertex locations of the swept fluxes are determined by the orthogonal swept fluxes (illustrated in Figs. 3a–c). The upstream vertex locations stay fixed. We then introduce an extra degree of freedom along the flux side (which coincides with a coordinate line crossing else it is halfway in between the upstream vertices) and move that point (along the coordinate line) to make the SE and CSLAM fluxes match. Since the vertex locations stay fixed, the iteration only affects the swept flux area for the side in question. Therefore, the iteration algorithm is local and constrained on each side by the prescribed SE flux. As illustrated in Fig. 7, the union of the swept flux areas and the Eulerian cell define the equivalent upstream Lagrangian area. This equivalent upstream Lagrangian grid is shown in Fig. 7b for the new CSLAM algorithm. Note that had we chosen to perturb the departure points (Fig. 7a), then a global iteration problem would result since the departure points are shared between Lagrangian cells (or equivalently swept flux areas). This global iteration problem would also require intersections between Eulerian and Lagrangian cells to be computed during each iteration, which is an expensive operation. For example, if the global iteration problem would converge in 10 iterations, the cost (in the current implementation of the original CSLAM) would be equivalent to transporting approximately 90–140 tracers with the SE method (not shown).
A graphical illustration of two different ways to make CSLAM and SE consistent. The dark blue filled circles are the departure points, the dark blue lines are the sides of the Lagrangian CSLAM cells, and the black lines are the CSLAM control volumes. The arrows indicate the points that are perturbed to make SE and CSLAM consistent. (a) SE and CSLAM are made consistent by perturbing the departure points. Note that this problem is a global iteration problem requiring computation of overlap areas during each iteration. (b) The approach chosen in this paper. The intersections between the sides of the Lagrangian cells with the grid lines are perturbed. This constitutes a local iteration problem as fluxes are matched with SE fluxes for each flux side, whereas the departure points stay fixed.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
A graphical illustration of two different ways to make CSLAM and SE consistent. The dark blue filled circles are the departure points, the dark blue lines are the sides of the Lagrangian CSLAM cells, and the black lines are the CSLAM control volumes. The arrows indicate the points that are perturbed to make SE and CSLAM consistent. (a) SE and CSLAM are made consistent by perturbing the departure points. Note that this problem is a global iteration problem requiring computation of overlap areas during each iteration. (b) The approach chosen in this paper. The intersections between the sides of the Lagrangian cells with the grid lines are perturbed. This constitutes a local iteration problem as fluxes are matched with SE fluxes for each flux side, whereas the departure points stay fixed.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
A graphical illustration of two different ways to make CSLAM and SE consistent. The dark blue filled circles are the departure points, the dark blue lines are the sides of the Lagrangian CSLAM cells, and the black lines are the CSLAM control volumes. The arrows indicate the points that are perturbed to make SE and CSLAM consistent. (a) SE and CSLAM are made consistent by perturbing the departure points. Note that this problem is a global iteration problem requiring computation of overlap areas during each iteration. (b) The approach chosen in this paper. The intersections between the sides of the Lagrangian cells with the grid lines are perturbed. This constitutes a local iteration problem as fluxes are matched with SE fluxes for each flux side, whereas the departure points stay fixed.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
4. Results
To assess the accuracy and efficiency of the CAM-SE–CSLAM algorithm, idealized baroclinic wave simulations have been performed with inert tracer distributions as well as idealized reactive chemical species. We compare the SE advection algorithm with the CSLAM advection algorithm. Note that the two transport schemes are driven by the same winds and airmass fields from the CAM-SE solution to the equations of motion. (The tests have been performed with the https://svn-homme-model.cgd.ucar.edu/branches/cslam@5168 code base.) This model version uses a floating Lagrangian vertical coordinate (Starr 1945; Lin 2004) but is otherwise equivalent to the CAM-SE version documented in Neale et al. (2012).
The model is run with 30 vertical levels (level locations as in CAM5), and the horizontal resolution is approximately 1° (
The CSLAM scheme has been implemented within the element structure (Erath et al. 2012) where each element has been subdivided into 
The idealized baroclinic wave used here is defined in Ullrich et al. (2014). This wave is similar to the Jablonowski and Williamson (2006) wave but with analytic support for deep atmosphere approximations. Both waves consist of a steady-state solution to the equations of motion with a perturbation that triggers the growth of a baroclinic wave. For this study, either of the baroclinic waves could have been used. In separate sections below, we describe the inert tracer transport test and reactive chemical species test, both being driven by the same SE solution to the Ullrich et al. (2014) baroclinic wave. For simplicity, we assume that the tracers are dimensionless.
a. Idealized baroclinic wave with inert tracers
(columns from left to right) 1° SE, 1° CSLAM, and ¼° SE solutions for tracer one 
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
(columns from left to right) 1° SE, 1° CSLAM, and ¼° SE solutions for tracer one
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
(columns from left to right) 1° SE, 1° CSLAM, and ¼° SE solutions for tracer one
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
As in Fig. 8, but for tracer two 
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
As in Fig. 8, but for tracer two
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
As in Fig. 8, but for tracer two
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
As in Fig. 8, but for tracer three 
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
As in Fig. 8, but for tracer three
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
As in Fig. 8, but for tracer three
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
The wind and surface pressure evolution at these two resolutions are significantly different when the baroclinic wave starts growing and rapidly evolves with explosive cyclogenesis between model days 6 and 9. This is illustrated in the 
Standard 
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
Standard
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
Standard
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
Figure 11b shows 
For the zonally symmetric distribution 
In the case of the slotted-cylinder initial distribution 
b. Idealized baroclinic wave with reactive chemical species
Initial condition for (top left) 
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
Initial condition for (top left)
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
Initial condition for (top left)
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
As in Fig. 12, but for day 15 using CSLAM.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
As in Fig. 12, but for day 15 using CSLAM.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
As in Fig. 12, but for day 15 using CSLAM.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
As in Fig. 12, but for day 15 using SE.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
As in Fig. 12, but for day 15 using SE.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
As in Fig. 12, but for day 15 using SE.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
c. Computational performance
Computational performance experiments are performed on the NCAR Yellowstone cluster (Computational and Information Systems Laboratory 2012) with standard computer optimization flags. We do not make use of threading. The amount of information that needs to be communicated between elements (if residing on different tasks) is very different between SE and CSLAM—similarly for the frequency of communication between elements. While the SE method only needs to share values at quadrature points located on the boundary of an element, communication is performed at every Runge–Kutta stage and every subcycling of the hyperviscosity operator. For the 1° model, that amounts to 21 communications per remapping time step (900 s), 5 communications per 300-s tracer time step, and 2 communications for hyperviscosity per tracer time step. CSLAM only needs one communication per 900-s tracer time step. That said, CSLAM needs all finite-volume values for each element surrounding the element being updated—that is, 72 values per tracer in the horizontal compared to 20 values for SE (four points on each edge plus the corner points). Figure 15a shows throughput for tracers for a 1-day simulation using SE (purple and black) and CSLAM (green) as a function of the number of tracers. The throughput for SE is shown for the CESM1 release code (purple line). The CAM-SE tracer advection code is currently being optimized further (J. Dennis 2016, personal communication). At a relatively low tracer count (at least for climate and climate-chemistry modeling), CSLAM is faster than SE. Note that the new CSLAM algorithm does not make use of an extensive search algorithm as the original formulation of CSLAM. The crossover between SE and CSLAM with the old formulation of flux-form CSLAM in HOMME (Harris et al. 2011) is over 20 tracers (not shown) and with the new consistent CSLAM algorithm, it is closer to 10 tracers. Figure 15a also shows the breakdown of computational cost for the different stages of the CSLAM algorithm. The departure point algorithm (orange line) and computation of high-order weights (yellow line) remains constant as a function of tracers and makes up a small cost even at low tracer counts. The communication (light blue), reconstruction (dark blue), and remapping (multiplication of reconstruction coefficients and precomputed weights; red) are all a function of tracers. The computation of the reconstruction coefficients is the most expensive step, mostly because of the reconstruction limiter that involves 
(a) Time (s) spent in various parts of the CSLAM transport algorithm for a 1-day simulation using 256 tasks as a function of number of tracers (the yellow line is on top of the orange line). The throughput for SE transport based on the CESM1 release (CAM-SE) is also shown. See text for details. (b) As in (a) but for a fixed number of tracers (40) and as a function of tasks. Note that for the rightmost data point (5400 tasks), there is only one element in the horizontal per processor. This is the limit of scalability for CAM-SE and CAM-SE–CSLAM without threading.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
(a) Time (s) spent in various parts of the CSLAM transport algorithm for a 1-day simulation using 256 tasks as a function of number of tracers (the yellow line is on top of the orange line). The throughput for SE transport based on the CESM1 release (CAM-SE) is also shown. See text for details. (b) As in (a) but for a fixed number of tracers (40) and as a function of tasks. Note that for the rightmost data point (5400 tasks), there is only one element in the horizontal per processor. This is the limit of scalability for CAM-SE and CAM-SE–CSLAM without threading.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
(a) Time (s) spent in various parts of the CSLAM transport algorithm for a 1-day simulation using 256 tasks as a function of number of tracers (the yellow line is on top of the orange line). The throughput for SE transport based on the CESM1 release (CAM-SE) is also shown. See text for details. (b) As in (a) but for a fixed number of tracers (40) and as a function of tasks. Note that for the rightmost data point (5400 tasks), there is only one element in the horizontal per processor. This is the limit of scalability for CAM-SE and CAM-SE–CSLAM without threading.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0258.1
Especially for lower-resolution modeling, strong scaling is important. Figure 15b shows throughput as a function of tasks/processors. Note that the last data point on the right-hand side of the plot (5400 processors) is the throughput where there is only one element in the horizontal per processor. It is well known that SE exhibits strong scaling; however, CSLAM also scales to the limit of scalability despite the large halo.
Compared to the original formulation of CSLAM, the new CSLAM algorithm does not make use of an extensive search algorithm. It has been replaced with a local iterative algorithm to enforce consistency with SE. The communication structure (amount of data communicated and the frequency of communication), reconstruction, shape-preserving limiter, and remapping algorithm are essentially the same as the original CSLAM scheme. Hence, the detailed performance analysis as a function of cores and tracers presented in Erath et al. (2012) is also relevant for the algorithm presented here.
CAM-SE–CSLAM has been presented in the context of a dry dynamical core. The extension to a full physics setup requires several modifications to the existing modeling system. First of all, one need to make choices about which grid is used to run the physics parameterizations (quadrature grid or CSLAM control volumes). In either case, tendencies from the parameterizations must be mapped between grids. Second, the treatment of moisture in the dynamical core can complicate the coupling between CAM-SE and CSLAM. Currently, CAM-SE is based on a vertical coordinate based on moist (full) surface pressure, which implies that the model levels move when there are changes in the moisture field from parameterized sources/sinks of moisture. These sources/sinks need to be accounted for both in SE and CSLAM. To avoid this, dry-mass vertical coordinates, in which the model levels are determined by dry surface pressure (as well as the usual hybrid coefficients), can be used. With dry-mass vertical coordinates, the model levels do not change because of parameterized water vapor sources/sinks. The infrastructure to run physics, tracers, and dynamics on different grids has been added to CAM, and a dry-mass vertical coordinate version of CAM-SE has been derived and implemented. This version also easily accounts for condensate loading. Details on these developments are beyond the scope of this paper.
5. Summary
In this paper, it has been shown that it is possible to consistently couple CAM-SE with the CSLAM transport algorithm without violating desirable properties such as shape preservation, mass conservation, linear correlation preservation, and consistency (aka compatibility, free-stream preservation). This is achieved by first deriving formulas for diagnosing SE airmass flux through the CSLAM control volume faces. These mass fluxes are used to construct CSLAM swept areas for which the airmass fluxes computed with CSLAM match the SE airmass fluxes to round off. The iterative algorithm is local and results in an equivalent upstream Lagrangian CSLAM grid that spans the sphere without gaps or overlaps. This new consistent SE-CSLAM algorithm avoids the expensive search for overlap areas used in the original formulations of CSLAM. The source code for computing weights has almost been cut in half with the new algorithm compared to the original CSLAM. This is mainly due to the new algorithm being formulated more generally, so special case statements for the edges and corners of the sphere have been reduced to a minimum.
It has been shown in idealized tracer experiments that CSLAM is more accurate than SE for challenging tracer distributions and at the same time is more efficient that SE if one is using more than approximately 12–15 tracers. Also, the CAM-SE–CSLAM model retains the excellent strong scaling properties of CAM-SE. The algorithm for making the CSLAM Lagrangian grid consistent with SE is general in the sense that CSLAM can be made consistent with any airmass fluxes as long as the fluxes satisfy the Lipshitz criterion.
Acknowledgments
Steve Goldhaber, James Overfelt, Paul Ullrich, and Mark Taylor were supported by the U.S. Department of Energy Office of Biological and Environmental Research, Work Package 12-015334 “Multiscale Methods for Accurate, Efficient, and Scale-Aware Models of the Earth System.”
APPENDIX
Spectral Element Subcell Flux
Our goal is to write each of the three terms on the right-hand side of (8) as a sum of edge fluxes in such a way that the fluxes along an edge shared by two subcells will be equal in magnitude and of opposite sign.
a. Strong form divergence: 
We note that in the SE method, at an edge shared by neighboring elements, both J, 
b. Weak form diffusion: 
This term, where 
For each subinterval, we need to decompose the change in mass into left, right, top, and bottom fluxes: 
Since 
c. DSS operation: 
The definition of 
REFERENCES
Barth, T. J., and D. C. Jespersen, 1989: The design and application of upwind schemes on unstructured meshes. Proc. AIAA 27th Aerospace Sciences Meeting, Reno, NV, American Institute of Aeronautics and Astronautics, 1–12, doi:10.2514/6.1989-366.
Computational and Information Systems Laboratory, 2012: Yellowstone: IBM iDataPlex system. NCAR/UCAR. [Available online at http://n2t.net/ark:/85065/d7wd3xhc.]
Dawson, C., 1999: Conservative, shock-capturing transport methods with nonconservative velocity approximations. Comput. Geosci., 3, 205–227, doi:10.1023/A:1011539311766.
Deleersnijder, E., and G. Lebon, 2001: Enforcing the continuity equation in numerical models of geophysical fluid flows. Appl. Math. Lett., 14, 867–873, doi:10.1016/S0893-9659(01)00057-X.
Dennis, J., A. Fournier, W. Spotz, A. St-Cyr, M. A. Taylor, S. J. Thomas, and H. Tufo, 2005: High-resolution mesh convergence properties and parallel efficiency of a spectral element atmospheric dynamical core. Int. J. High Perform. Comput. Appl., 19, 225–235, doi:10.1177/1094342005056108.
Dennis, J., and Coauthors, 2012: CAM-SE: A scalable spectral element dynamical core for the Community Atmosphere Model. Int. J. High. Perform. Comput. Appl., 26, 74–89, doi:10.1177/1094342011428142.
Deville, M. O., P. F. Fischer, and E. H. Mund, 2002: High-Order Methods for Incompressible Fluid Flow. Cambridge Monographs on Applied and Computational Mathematics, No. 9, Cambridge University Press, 528 pp.
Erath, C., and R. D. Nair, 2014: A conservative multi-tracer transport scheme for spectral-element spherical grids. J. Comput. Phys., 256, 118–134, doi:10.1016/j.jcp.2013.08.050.
Erath, C., P. H. Lauritzen, J. H. Garcia, and H. M. Tufo, 2012: Integrating a scalable and efficient semi-Lagrangian multi-tracer transport scheme in HOMME. Procedia Comput. Sci., 9, 994–1003, doi:10.1016/j.procs.2012.04.106.
Erath, C., P. H. Lauritzen, and H. M. Tufo, 2013: On mass conservation in high-order high-resolution rigorous remapping schemes on the sphere. Mon. Wea. Rev., 141, 2128–2133, doi:10.1175/MWR-D-13-00002.1.
Gross, E. S., L. Bonaventura, and G. Rosatti, 2002: Consistency with continuity in conservative advection schemes for free-surface models. Int. J. Numer. Methods Fluids, 38, 307–327, doi:10.1002/fld.222.
Guba, O., M. Taylor, and A. St-Cyr, 2014: Optimization-based limiters for the spectral element method. J. Comput. Phys., 267, 176–195, doi:10.1016/j.jcp.2014.02.029.
Hall, D. M., P. A. Ullrich, K. A. Reed, C. Jablonowski, R. D. Nair, and H. M. Tufo, 2016: Dynamical Core Model Intercomparison Project (DCMIP) tracer transport test results for CAM-SE. Quart. J. Roy. Meteor. Soc., 142, 1672–1684, doi:10.1002/qj.2761.
Harris, L. M., P. H. Lauritzen, and R. Mittal, 2011: A flux-form version of the Conservative Semi-Lagrangian Multi-tracer transport scheme (CSLAM) on the cubed sphere grid. J. Comput. Phys., 230, 1215–1237, doi:10.1016/j.jcp.2010.11.001.
Jablonowski, C., and D. L. Williamson, 2006: A baroclinic instability test case for atmospheric model dynamical cores. Quart. J. Roy. Meteor. Soc., 132, 2943–2975, doi:10.1256/qj.06.12.
Kuo, H.-C., and R. T. Williams, 1990: Semi-Lagrangian solutions to the inviscid Burgers equation. Mon. Wea. Rev., 118, 1278–1288, doi:10.1175/1520-0493(1990)118<1278:SLSTTI>2.0.CO;2.
Lauritzen, P. H., and J. Thuburn, 2012: Evaluating advection/transport schemes using interrelated tracers, scatter plots and numerical mixing diagnostics. Quart. J. Roy. Meteor. Soc., 138, 906–918, doi:10.1002/qj.986.
Lauritzen, P. H., C. Jablonowski, M. A. Taylor, and R. D. Nair, 2010a: Rotated versions of the Jablonowski steady-state and baroclinic wave test cases: A dynamical core intercomparison. J. Adv. Model. Earth Syst., 2, 15, doi:10.3894/JAMES.2010.2.15.
Lauritzen, P. H., R. D. Nair, and P. A. Ullrich, 2010b: A Conservative Semi-Lagrangian Multi-tracer transport scheme (CSLAM) on the cubed-sphere grid. J. Comput. Phys., 229, 1401–1424, doi:10.1016/j.jcp.2009.10.036.
Lauritzen, P. H., P. A. Ullrich, and R. D. Nair, 2011: Atmospheric transport schemes: Desirable properties and a semi-Lagrangian view on finite-volume discretizations. Numerical Techniques for Global Atmospheric Models, P. H. Lauritzen et al., Eds., Lecture Notes in Computational Science and Engineering, Vol. 80, Springer, 185–250, doi:10.1007/978-3-642-11640-7_8.
Lauritzen, P. H., W. C. Skamarock, M. J. Prather, and M. A. Taylor, 2012: A standard test case suite for two-dimensional linear transport on the sphere. Geosci. Model Dev., 5, 887–901, doi:10.5194/gmd-5-887-2012.
Lauritzen, P. H., J. T. Bacmeister, T. Dubos, S. Lebonnois, and M. A. Taylor, 2014a: Held–Suarez simulations with the Community Atmosphere Model spectral element (CAM-SE) dynamical core: A global axial angular momentum analysis using Eulerian and floating Lagrangian vertical coordinates. J. Adv. Model. Earth Syst., 6, 129–140, doi:10.1002/2013MS000268.
Lauritzen, P. H., and Coauthors, 2014b: A standard test case suite for two-dimensional linear transport on the sphere: Results from a collection of state-of-the-art schemes. Geosci. Model Dev., 7, 105–145, doi:10.5194/gmd-7-105-2014.
Lauritzen, P. H., A. J. Conley, J.-F. Lamarque, F. Vitt, and M. A. Taylor, 2015: The terminator “toy” chemistry test: A simple tool to assess errors in transport schemes. Geosci. Model Dev., 8, 1299–1313, doi:10.5194/gmd-8-1299-2015.
Lin, S.-J., 2004: A “vertically Lagrangian” finite-volume dynamical core for global models. Mon. Wea. Rev., 132, 2293–2307, doi:10.1175/1520-0493(2004)132<2293:AVLFDC>2.0.CO;2.
Lin, S.-J., and R. B. Rood, 1996: Multidimensional flux-form semi-Lagrangian transport schemes. Mon. Wea. Rev., 124, 2046–2070, doi:10.1175/1520-0493(1996)124<2046:MFFSLT>2.0.CO;2.
Lopez, N. A., and M. A. Taylor, 2015: Improving the tracer correlation problem in a spectral element dynamical core. Center for Computing Research Summer Proc. 2015, Albuquerque, NM, Sandia National Laboratories, 106–117. [Available online at https://cfwebprod.sandia.gov/cfdocs/CompResearch/docs/proceedings/ccr15.pdf#page=112.]
Nair, R. D., and P. H. Lauritzen, 2010: A class of deformational flow test cases for linear transport problems on the sphere. J. Comput. Phys., 229, 8868–8887, doi:10.1016/j.jcp.2010.08.014.
Nair, R. D., J. S. Scroggs, and F. H. M. Semazzi, 2002: Efficient conservative global transport schemes for climate and atmospheric chemistry models. Mon. Wea. Rev., 130, 2059–2073, doi:10.1175/1520-0493(2002)130<2059:ECGTSF>2.0.CO;2.
Nair, R. D., H.-W. Choi, and H. M. Tufo, 2009: Computational aspects of a scalable high-order discontinuous Galerkin atmospheric dynamical core. Comput. Fluids, 38, 309–319, doi:10.1016/j.compfluid.2008.04.006.
Neale, R. B., and Coauthors, 2012: Description of the NCAR Community Atmosphere Model (CAM 5.0). NCAR Tech. Note NCAR/TN-486+STR, 274 pp. [Available online at http://www.cesm.ucar.edu/models/cesm1.0/cam/docs/description/cam5_desc.pdf.]
Pudykiewicz, J., R. Benoit, and A. Staniforth, 1985: Preliminary results from a partial LRTAP model based on an existing meteorological forecast model. Atmos.–Ocean, 23, 267–303, doi:10.1080/07055900.1985.9649229.
Putman, W. M., and S.-J. Lin, 2007: Finite-volume transport on various cubed-sphere grids. J. Comput. Phys., 227, 55–78, doi:10.1016/j.jcp.2007.07.022.
Rančić, M., R. J. Purser, and F. Mesinger, 1996: A global shallow-water model using an expanded spherical cube: Gnomonic versus conformal coordinates. Quart. J. Roy. Meteor. Soc., 122, 959–982, doi:10.1002/qj.49712253209.
Rood, R. B., 1987: Numerical advection algorithms and their role in atmospheric transport and chemistry models. Rev. Geophys., 25, 71–100, doi:10.1029/RG025i001p00071.
Shashkin, V. V., and M. A. Tolstykh, 2016: Parallel implementation of the cascade mass-conserving semi-Lagrangian transport scheme. Russ. J. Numer. Anal. Math. Modell., 31, 17–28, doi:10.1515/rnam-2016-0002.
Starr, V. P., 1945: A quasi-Lagrangian system of hydrodynamical equations. J. Meteor., 2, 227–237, doi:10.1175/1520-0469(1945)002<0227:AQLSOH>2.0.CO;2.
St-Cyr, A., C. Jablonowski, J. M. Dennis, H. M. Tufo, and S. J. Thomas, 2008: A comparison of two shallow-water models with nonconforming adaptive grids. Mon. Wea. Rev., 136, 1898–1922, doi:10.1175/2007MWR2108.1.
Taylor, M. A., 2011: Conservation of mass and energy for the moist atmospheric primitive equations on unstructured grids. Numerical Techniques for Global Atmospheric Models, P. H. Lauritzen et al. Eds., Lecture Notes in Computational Science and Engineering, Vol. 80, Springer, 357–380, doi:10.1007/978-3-642-11640-7_12.
Taylor, M. A., and A. Fournier, 2010: A compatible and conservative spectral element method on unstructured grids. J. Comput. Phys., 229, 5879–5895, doi:10.1016/j.jcp.2010.04.008.
Taylor, M. A., J. Tribbia, and M. Iskandarani, 1997: The spectral element method for the shallow water equations on the sphere. J. Comput. Phys., 130, 92–108, doi:10.1006/jcph.1996.5554.
Ullrich, P. A., P. H. Lauritzen, and C. Jablonowski, 2009: Geometrically exact conservative remapping (GECoRe): Regular latitude–longitude and cubed-sphere grids. Mon. Wea. Rev., 137, 1721–1741, doi:10.1175/2008MWR2817.1.
Ullrich, P. A., T. Melvin, C. Jablonowski, and A. Staniforth, 2014: A proposed baroclinic wave test case for deep- and shallow-atmosphere dynamical cores. Quart. J. Roy. Meteor. Soc., 140, 1590–1602, doi:10.1002/qj.2241.
Zarzycki, C. M., M. N. Levy, C. Jablonowski, J. R. Overfelt, M. A. Taylor, and P. A. Ullrich, 2014: Aquaplanet experiments using CAM’s variable-resolution dynamical core. J. Climate, 27, 5481–5503, doi:10.1175/JCLI-D-14-00004.1.
Zerroukat, M., N. Wood, and A. Staniforth, 2002: SLICE: A semi-Lagrangian inherently conserving and efficient scheme for transport problems. Quart. J. Roy. Meteor. Soc., 128, 2801–2820, doi:10.1256/qj.02.69.
