## 1. Introduction

Because of the changing trends in high-performance computers, from large vector machines to distributed-memory architectures, numerical methods that decompose the physical domain into smaller pieces have been receiving significant attention. This new focus on local methods is especially true in the atmospheric sciences, where very large models covering the entire globe are run in timescales ranging from days (in numerical weather prediction) to thousands of years (in climate simulations). Finite-difference and finite-element methods are two such methods that decompose the domain locally, thereby facilitating their implementation on distributed-memory computers. However, one of the biggest disadvantages of these methods is that traditionally they have not been able to compete, in terms of accuracy, with spectral transform methods, which are typically used operationally in numerical weather prediction (NWP) and climate modeling. For example, spectral transform models are used by the National Centers for Environmental Prediction (NCEP; Sela 1980), the European Centre for Medium-Range Weather Forecasts (ECMWF; Simmons et al. 1989), the National Center for Atmospheric Research (NCAR; Hack et al. 1992), and the U.S. Navy (Hogan and Rosmond 1991).

Spectral element methods combine the local domain decomposition property of finite-element methods with the high-order accuracy of spectral transform methods. In other words, spectral elements are as local as finite-element methods and thereby can be used as efficiently on distributed-memory computers while sustaining the same level of accuracy obtained with spectral transform methods. Spectral element methods have been used successfully for the shallow water equations on the sphere (Giraldo 2001; Giraldo et al. 2002; Taylor et al. 1997) and have shown to be promising for ocean and climate modeling (Iskandarani et al. 2002; Loft et al. 2001; Thomas et al. 2002). These methods are essentially high-order finite-element methods where the grid points are chosen to be the Legendre–Gauss–Lobatto (LGL) points. This choice of grid points allows for stable high-order interpolations and results in efficient numerical integration strategies because the LGL points are also used as the quadrature points in the numerical integration required by the weak integral formulation common to all Galerkin methods.

*λ,*

*φ*), of radius

*a,*the divergence of a vector field,

**F**=

*f*

*λ̂**g*

**φ̂**

*φ*= ±

*π*/2), this is a source of numerical problems, caused by the specific coordinate formulation rather than the nature of the primitive equations and its solutions. While the use of a local Cartesian coordinate system has been used to overcome these problems in the past (Taylor et al. 1997), we have, guided by the results of previous work (Giraldo 2001; Giraldo et al. 2002), chosen to maintain the Cartesian formulation everywhere.

Therefore, in our formulation the poles are treated as any other point in Cartesian space. Because the numerical method is constructed independently of the grid, this then implies that any grid can be used within this framework including icosahedral, hexahedral, lat–lon, and adaptive unstructured grids. The option of using adaptive unstructured grids will facilitate the coupling of this dynamical core with the Naval Research Laboratory's (NRL) mesoscale model (Hodur 1997). The independence of our numerical methodology from the grid also means that we can change the basis functions from continuous to discontinuous, as we showed in Giraldo et al. (2002), or the elements on which these functions are constructed from quadrilaterals to triangles (Warburton et al. 2000), which then simplifies the construction of adaptive solutions. This independence from the grid is not shared by any of the existing and newly proposed global atmospheric models, including the spectral element model in Taylor et al. (1997), Loft et al. (2001), and Thomas et al. (2002) and the icosahedral model in Randall et al. (2002). In fact, the formulations of all these models are bounded on a specific class of grids. Furthermore, the Cartesian formulation simplifies the addition of semi-Lagrangian schemes. In this paper, we refer to our current model as *Eulerian* in order to distinguish it from the semi-Lagrangian version we are currently testing in other work. In brief, the objective of this paper is to show the feasibility of the Cartesian spectral element formulation for constructing hydrostatic primitive equation models that are as accurate as current spectral transform models and more efficient on distributed-memory computers.

The remainder of the paper is organized as follows. Section 2 contains a description of the governing equations of motion used in numerical weather prediction models, along with a detailed definition of the prognostic and diagnostic variables used in our model. Section 3 contains the description of the numerical approximation of the equations, including, the horizontal, vertical, and temporal discretization methods. In section 4 we describe the tessellation of the sphere into the quadrilateral elements used by the spectral element method to construct the local element matrix operations. This leads directly into section 5, which contains a discussion on the domain decomposition of the sphere and how it translates into the implementation of the model on distributed-memory computers using the Message-Passing Interface. In section 6 we present the results for the seven test cases used to validate our model. Finally, in section 7 we summarize the key findings of this research and discuss the direction of future work.

## 2. Atmospheric equations

*π,*the three Cartesian velocity components,

**u**= (

*u,*

*υ,*

*w*), and the potential temperature,

*θ.*The diagnostic variables are the vertical velocity

*σ̇*

*p,*and geopotential height

*ϕ.*

*a*and

*ω*are the earth's radius and angular velocity, and

*μ*is a Lagrange multiplier used to constrain the fluid particles to remain on each spherical shell defined by the vertical coordinate

*σ*[we shall describe the role of the Lagrange multiplier in detail in section 3d(1)]. The independent variables in this coordinate system are (

*x,*

*y,*

*z,*

*σ,*

*t*), where the triple (

*x,*

*y,*

*z*) represents the grid point on the sphere defined by the spherical coordinates (

*λ,*

*φ*) and are related by

*σ.*

*π,*in the governing equations is defined as

*π*

*p*

_{s}

*p*

_{t}

*p*

_{s}is the true surface pressure, and

*p*

_{t}is the pressure at the top of the atmosphere. The potential temperature

*θ*is defined as

*T*is the temperature, and

*P*is the Exner function. The Exner function is defined as

*κ*is the air constant

*R*

_{d}/

*c*

_{p},

*R*

_{d}= 287 J/kg K is the gas constant of dry air,

*c*

_{p}= 1004 J/kg K is the specific heat of dry air at constant pressure, and

*p*

_{o}= 1000 hPa is the standard surface pressure. The vertical velocity is defined as

*σ̇*

*dσ*/

*dt,*where

*σ*is given by

*d*/

*dt*is the Lagrangian derivative.

## 3. The numerical scheme

To solve Eqs. (1), (2), and (3), we split the spatial operators into their horizontal and vertical components. Therefore for a given *σ* value, we discretize the horizontal operators defined on a constant *σ* spherical shell, as was done in Giraldo (2001) using the spectral element method. The vertical operators are discretized by a mass- and energy-conserving flux-form finite-difference method. We begin with the horizontal discretization of the equations by the spectral element method.

### a. Approximating the solution in the horizontal direction

#### 1) Basis functions and integration

**Ω**into

*N*

_{e}nonoverlapping quadrilateral elements such that

To perform differentiation and integration operations, we introduce the nonsingular mapping **x** = ϒ(** ξ**), which defines a transformation from the physical Cartesian coordinate system

**x**= (

*x,*

*y,*

*z*) defined in

**Ω**

_{e}to the reference coordinate system

**= (**

*ξ**ξ,*

*η,*

*ζ*) defined in each element, where (

*ξ,*

*η*) ∈ [−1, +1]

^{2}in each element, and

*ζ*= 1 on the surface of the sphere.

*J*= ∂

**x**/∂

**, and the determinant**

*ξ***G**represents the surface-conforming component of the mapping [see Giraldo (2001) for further details].

We can now use this mapping to define the local representation of the solution, **q** = (*π,* **u**, *θ*), and the approximation of operations such as differentiation and integration. For simplicity, we assume *ζ* to be unity in what remains and denote ** ξ** = (

*ξ,*

*η*).

**∈ [−1, 1]**

*ξ*^{2}, makes it natural to represent the local elementwise solution

**q**by an

*N*th-order polynomial in

**as**

*ξ***x**

_{k}represents (

*N*+ 1)

^{2}grid points, and

*ψ*

_{k}(

**x**) is the associated multivariate Lagrange polynomial. The logical square structure of 𝗜 simplifies matters in that we can express the Lagrange polynomial by a tensor product as

*ψ*

_{k}

**x**

*h*

_{i}

*ξ*

**x**

*h*

_{j}

*η*

**x**

*i,*

*j*= 0, … ,

*N,*and

*k*= 1, … , (

*N*+ 1)

^{2}. In Eq. (6),

*h*are the one-dimensional Lagrange polynomials

*P*

_{N}(

*ξ*) is the

*N*th-order Legendre polynomial. For the grid points (

*ξ*

_{i},

*η*

_{j}), we choose the LGL points, given as the tensor product of the roots of

*ξ*

^{2}

*P*

^{′}

_{N}

*ξ*

**Ω**

_{e}and 𝗜, and

*ω*(

*ξ*

_{i}) and

*ω*(

*η*

_{j}) are the Gaussian quadrature weights,

**F**represents the flux tensor, and

**S**represents the source terms that we define explicitly in the appendix. Taking the weak form of Eq. (7) with respect to global basis functions Ψ gives

**q**and

**F**by the global polynomial approximation similar to Eq. (5) yields the global Galerkin projection of the governing equations

*I,*𝗝 = 1, … ,

*N*

_{p}, with

*N*

_{p}representing the number of grid points in the horizontal. We shall return to the discussion on the construction of the global solution in section 3a(2). Because the global operators given in Eq. (9) are never explicitly defined, we begin by defining the local operators that are in fact constructed and then used to construct the action of these global operators on the state vector.

*ψ*(

**x**) are the local element basis functions given in Eq. (6),

*i,*

*j*= 1, … , (

*N*+ 1)

^{2}are the number of grid points within each element

**Ω**

_{e}, and 𝗗 = (𝗗

^{x}, 𝗗

^{y}, 𝗗

^{z}) is a vector of matrices corresponding to the three spatial directions. The role of these local element matrices are described below.

#### 2) Satisfying the equations globally

*E*1, … ,

*E*4), and then each element contributes its local approximation to the global sum. For example, the element matrix contributions at the local grid points 4, 3, 2, and 1 of elements E1, E2, E3, and E4 are summed in order to construct the value of the global matrix at the global grid point G1. It should be understood that the local grid points (4, 3, 2, 1) are the same grid points that are claimed by different elements (and possibly processors), and which is G1 in the global indexing. Let us represent this global assembly procedure by the summation operator

*i,*

*e*) → (

*I*), where

*i*= 1, … , (

*N*+ 1)

^{2}are the local element grid points,

*e*= 1, … ,

*N*

_{e}are the spectral elements covering the spherical shell, and

*I*= 1, … ,

*N*

_{p}are the global grid points. Applying the global assembly operator to the local element matrices results in the following global matrices:

*π*

_{G}, the wind velocity as

**u**

_{G}, the potential temperature as

*θ*

_{G}, and the geopotential height as

*ϕ*

_{G}, we can now write the semidiscrete approximation to Eqs. (1), (2), and (3) as follows:

_{G}denote the global grid vector of the quantities inside the brackets after they have been vertically discretized. It should be noted that the mass matrix,

*M,*is diagonal and thereby trivial to invert. The diagonal property of this matrix is due to the dual role of the LGL points, which are used both as grid points and quadrature points. Furthermore, the global matrix 𝗗 is never actually constructed, but rather only its action on the state vector

**q**is computed by virtue of the local element matrix and the global assembly procedure. We now address the discretization of the vertical operators.

### b. Approximating the solution in the vertical direction

The equations are discretized in the vertical direction using a conservative flux-form finite-difference method. This is the same vertical differencing method used in the Navy Operational Global Atmospheric Prediction System (NOGAPS), which is the U.S. Navy's current global atmospheric NWP model (Hogan and Rosmond 1991; Rosmond 2000). NOGAPS is used by the Navy for medium-range weather forecasts worldwide. This model is used to drive the Navy's mesoscale model (Hodur 1997) and is used as a coupled ocean–atmosphere system (Rosmond et al. 2002). NOGAPS uses the spectral transform method in the horizontal, a flux-form finite-difference in the vertical, and a semi-implicit leapfrog scheme in time. The horizontal resolution of NOGAPS recently increased from T159 with 24 vertical levels to T239 with 30 vertical levels (Hogan et al. 2002).

Although we could also discretize the vertical operators in the spectral element Eulerian atmospheric model (SEE-AM) with the spectral element method, we have chosen to use the finite-difference method in order to remain as similar as possible to NOGAPS. This will ensure that any differences in the results are due only to discrepancies in the discrete horizontal operators between the two models. We hope to report on a spectral element vertical discretization in future work.

**q**= (

_{G}*π*

_{G},

**u**,

_{G}*θ*

_{G}) to be

*K*denotes the number of vertical levels to be integrated across, and

*σ*

_{L}

*σ*

_{L+1/2}

*σ*

_{L−1/2}

*N*

_{lev}denotes the total number of vertical levels. Once the surface pressure tendency ∂

*π*

_{G}/∂

*t*is computed, the vertical velocity

*σ̇*

*σ*= 0) to the desired vertical level

*K,*thus giving

*K*denotes a full level and

*K*± 1/2 denote the half levels of this staggered sigma coordinate system. The prognostic variables and

*ϕ*all reside at the full levels, while the diagnostic variables reside at the half levels. The top and bottom of the atmosphere are at the half levels

*K*= 1/2 and

*K*=

*N*

_{lev}+ 1/2, respectively. Figure 2 illustrates the sigma coordinate system and the location of the prognostic and diagnostic variables.

*σ̇*

*σ*) of the global solution vector

**q**are computed in the following manner:

_{G}### c. Temporal discretization

_{max}is the maximum wave speed of the atmospheric equations (see the appendix for a derivation of this characteristic velocity). The physical spacing of the grid Δ

*s*scales as

*N*

_{e}is the number of spectral elements comprising the grid, and

*N*is the order of the polynomial approximation inside each element.

**q**at the end of each time step. Although this is not the most sophisticated time-discretization method available, we have used it in order to keep our model as similar to NOGAPS as possible. NOGAPS uses a semi-implicit time discretization, but this is applied as a correction to the explicit leapfrog scheme. Future research involves the addition of semi-implicit Eulerian and fully implicit semi-Lagrangian methods to the current formulation, which will hopefully allow an increase in the time step by a factor of 10.

_{G}### d. Lagrange multiplier and the high-pass filter

#### 1) Constraining the momentum

*σ*level), the fluid particles are allowed an extra degree of freedom. This degree of freedom will manifest itself in fluid particles flying off the spherical shell. Mollifying this undesirable situation requires constraining the velocity field to be tangential to the sphere. At each grid point we apply the following constraint

**u**

^{n+1}

_{c}

**u**

^{n+1}

_{u}

*μ*

**x**

*c*and

*u*denote the constrained and unconstrained horizontal wind velocities, respectively [see Côté (1988) for further details]. For a fluid particle to remain on the spherical shell, the wind velocity must be orthogonal to the position vector of its grid point; that is,

**u**

**x**

**u**

^{n+1}

_{c}

**u**

^{n+1}

_{u}

#### 2) High-pass filter

**q**in the

*ξ*direction as follows:

**q**

*L*

**q̃**

*ξ*

*P*

_{i}are the Nth-order Legendre polynomials, and

**q̃**are the Legendre modal coefficients. To filter the local solution

**q**, we transform them to modal space via the inverse of Eq. (31), apply the filter-weighting diagonal matrix

**Λ**and then transform back to nodal (gridpoint) space. This can be written as

**q**

^{1}

_{𝗙}

**q**

**Λ**

^{−1}

*N*+ 1) × (

*N*+ 1) matrix. The success of the filter hinges on the weighting matrix

**Λ**. Following the idea of P. F. Fischer (2002, personal communication), we write

*i*=

*i*

_{F}

*N*denotes the modes curtailed by the filter. In this paper we use

*i*

_{F}=

*N,*with

*μ*= 0.05, which represents only filtering the highest mode by 5%.

The reason for constructing the Legendre transform as *P*_{i} − *P*_{i−2} for *i* = 2, … , *N* and the weighting matrix **Λ** = 1 for *i* = 0, 1 is to avoid affecting the local element boundary values. By using this Legendre polynomial construction, only the first two modes (*P*_{0} and *P*_{1}) affect the element boundary values; the remainder of the modes only affect the element interior values. This way we can apply the filter in an element-by-element sense without violating the *C*^{0} continuity condition at the element interfaces (boundaries). This eliminates the need for global assembly of the local element filter matrix. The global assembly operation incurs communication costs on distributed-memory computers, and these costs must be minimized to achieve good performance.

^{e}

_{𝗙}

^{e}

^{T}

^{T}is the transpose of 𝗙, and 𝗾

^{e}is an (

*N*+ 1) × (

*N*+ 1) matrix containing the solution vector of the element

**Ω**

_{e}.

## 4. Grid generation on the sphere

One of the advantages of using Cartesian coordinates is that any grid can be used with our spectral element atmospheric model. Although we can use any grid whatsoever, at the moment the grids must be conforming and quadrilaterally shaped. Using the discontinuous basis functions as in Giraldo et al. (2002) will permit using nonconforming grids, and the spectral element basis in Warburton et al. (2000) will allow the use of triangles, which we reserve for future work. In order to show the grid independence of our model, in this paper we show results on icosahedral and hexahedral grids.

### a. Icosahedral grids

Icosahedral grids are constructed by subdividing the 20 triangular faces of the icosahedron by a Lagrange polynomial of order *n*_{I}, as described in Giraldo et al. (2002). Prior to mapping these elements onto the sphere, it is convenient to map the triangles onto a gnomonic space. The most unbiased mapping is obtained by mapping about the centroid of the triangles.

*λ*

_{c},

*φ*

_{c}) be the centroid of the triangle we wish to map. The gnomonic mapping is then given by

*R,*whereby Eq. (36) becomes

*λ,*

*φ*) located at (0, 0). The rotation mapping,

*R,*is defined as follows:

*N*

_{p}and

*N*

_{e}denote the total number of grid points and elements composing the icosahedral grid, and

*N*is the polynomial order of the elements. Examples of corresponding grids for

*n*

_{I}= 2,

*N*= 8 and

*n*

_{I}= 4,

*N*= 8 are illustrated in Fig. 3.

### b. Hexahedral grids

**G**, which is defined by the square region

**= [−**

*ξ*_{G}*π*/4, +

*π*/4]

^{2}in a 2D Cartesian space (Komatitsch and Tromp 2002; Ronchi et al. 1996). This region is divided into the elements and inside each element we construct the LGL grid points. Upon constructing this grid, we then map the gnomonic coordinates to the corresponding spherical coordinates,

**, via**

*λ*_{G}**only gives the spherical coordinates of one of the six faces of the hexahedron. Therefore, we have to rotate this face to the six faces of the hexahedron by the rotation mapping**

*λ*_{G}*R*

*λ*

_{c},

*φ*

_{c}), of the six faces are located at (

*λ*

_{c},

*φ*

_{c}) = ([

*c*− 1]

*π*/2, 0) for

*c*= 1, … , 4, and (

*λ*

_{5},

*φ*

_{5}) = (0,

*π*/2), (

*λ*

_{6},

*φ*

_{6}) = (0, −

*π*/2). This approach results in the construction of the hexahedral grid with the following properties:

*N*

_{p}and

*N*

_{e}denote the number of grid points and elements composing the grid. The parameter

*n*

_{H}refers to the number of quadrilateral elements in each direction (

*ξ*

_{G}and

*η*

_{G}) contained in each of the six faces of the hexahedron, and

*N*is the polynomial order of the elements. Figure 4 shows the grids

*n*

_{H}= 4,

*N*= 8 and

*n*

_{H}= 8,

*N*= 8. The hexahedral resolution

*H,*where

*H*

*n*

_{H}

*N,*

*T*on a Gaussian lat–lon grid; these two different grid resolutions are related by the expression

*H*

*T*

*N*

_{p}

*N*

_{lon}

*N*

_{lat}

*N*

^{2}

_{lat}

*N*

_{lon}and

*N*

_{lat}denote the number of points in the longitudinal and meridional directions in a Gaussian grid. In addition, let the spectral triangular truncation be given by

*T*= (2/3)

*N*

_{lat}− 1, which results in the number of grid points being

*H*∼ (

*T*+ 1), which we approximate by Eq. (44). Of course, there are other ways of relating the resolution of different grids but certainly using the number of grid points is the most logical approach. With this relationship between spectral triangular truncation,

*T,*and hexahedral resolution,

*H,*we can obtain the equivalent hexahedral resolution for the NOGAPS operational resolution of

*T*239 with

*n*

_{H}= 30 and

*N*= 8 (yielding

*H*240).

## 5. Parallel implementation

In this section we discuss the issues concerning the implementation of SEE-AM on distributed-memory computing platforms. Let us begin by describing the domain decomposition strategy.

### a. Domain decomposition

*N*

_{proc}

*n*

_{P}

^{2}

*N*

_{proc}is the total number of processors and

*n*

_{P}represents the partitioning of processors in each of the two Cartesian directions on each of the six faces of the hexahedron. In addition, we require the following constraint on

*n*

_{P}:

*n*

_{P}

*n*

_{H}

This is by no means the only possible partitioning strategy. A more sophisticated approach is to use the Metis graph partitioning software (Karypis and Kumar 1998) or the space-filling-curves strategy presented in Dennis (2003). We merely present this ad hoc hexahedral grid domain decomposition strategy as a proof of concept that SEE-AM performs efficiently on distributed-memory computers. In the future, we plan on implementing the space-filling-curves strategy, which will simplify the domain partitioning and allow us to use any grid whatsoever.

### b. Communication

From the description of the global assembly procedure given in section 3a(2), it should be apparent that the communication in the spectral element method results from the summation of the local element matrices to construct the global matrices. In order to better understand the communication that takes place across neighboring processors, let us look at only one face of the hexahedral grid. For the sake of argument, let us assume that *n*_{P} = 3, meaning that there are nine processors per face. This situation is illustrated in Fig. 5. Furthermore, let us assume that *n*_{H} = 9; that is, there are 81 elements per face. From the figure it is evident that there will then be nine elements inside each processor denoted by *N*1, … , *N*9 for the neighbors, and *P*1, … , *P*9 for the on-processor elements. The processors are denoted by the thick lines, while the thin lines represent the spectral elements. However, in order to keep the discussion as general as possible, we shall not define the order of the polynomial, *N,* inside each element.

Because of the *C*^{0} continuity condition required by the spectral element method, the four corner points of processor PROC in Fig. 5 are each shared by four processors: PROC plus three neighboring processors. Similarly, edge points are shared by two processors. This continuity condition is satisfied by the global assembly procedure that is executed across processors as follows. In processor PROC the on-processor right-hand-side (rhs) vectors of Eqs. (24), (26), and (27) are constructed. For the interior grid points of PROC, these rhs vectors represent the globally assembled rhs vectors of the equations; however, for the boundary grid points they are only a portion of the global rhs vectors. Thus, processor PROC requires the boundary gridpoint rhs vectors from processors NBH1, … , NBH8 to complete the global assembly of its rhs vectors. Once these global rhs vectors are constructed, each processor can then solve independently for the global solution via Eqs. (24), (26), and (27). Note that Eq. (25) is solved on-processor without communication after Eq. (24) is solved.

*N*

*N*

_{lev}

*α,*

*N,*

*N*

_{lev}, and

*α*; the constant in Eq. (46) arises from each processor having four edges and the primitive equations, in Cartesian coordinates, having five prognostic variables. Thus at every time step the perimeter values of the full 3D rhs vectors of each processor are sent to its neighbors.

The communication described above is exact for all processors that do not contain one of the eight corner points of the hexahedron. In Fig. 5, processor NBH5 only has seven neighboring processors to communicate with because of the topology of the hexahedron. In addition, for the special case *n*_{H} = 1, each processor has only four neighbors.

### c. Performance

#### 1) Model scalability

One of the main advantages of using spectral element methods over spectral transform methods is that for an equivalent resolution the spectral element method allows the use of far more processors. As an example let us compare SEE-AM with NOGAPS, which uses the spectral transform method in the horizontal. The most efficient decomposition for NOGAPS is through a 1D decomposition along latitude rings. It should be noted, however, that in general 2D decompositions are more efficient for spectral transform models, as shown in Foster et al. (1992). We shall only be comparing the operational version of NOGAPS with SEE-AM, and it should be understood that the discussion in this section does not necessarily extend to all spectral transform models, but it should provide some reasonable estimates.

*N*

_{lat}denotes the number of latitude rings, and

*T*the resolution of the spectral triangular truncation. In contrast, the maximum number of processors that SEE-AM can use is

*H.*In other words, SEE-AM can use as many processors as there are elements. Thus for fixed

*N*the number of processors allowed by SEE-AM increases quadratically with resolution,

*H,*while only linearly for NOGAPS. At the operational T239 resolution NOGAPS can use 360 processors, whereas SEE-AM (assuming

*n*

_{H}= 30 and

*N*= 8) can use 5400 processors; a 15-fold increase in the number of processors. Equation (50) shows that if we wish to further increase the number of processors with SEE-AM, we simply increase

*n*

_{H}while decreasing

*N*accordingly in order to maintain the horizontal resolution. Therefore we could use

*n*

_{H}= 60 and

*N*= 4 for a total of 21 600 processors; a 60-fold increase in the number of processors. However, decreasing

*N*will impact the solution accuracy and the issue of efficiency versus accuracy must be carefully weighed. The point here is that the spectral element method offers this flexibility to increase either the accuracy or efficiency—a luxury not shared by the spectral transform method. We leave the detailed discussion of the issue of efficiency versus accuracy to future work, but at this point we anticipate using

*N*= 8; even with

*N*= 8, SEE-AM accommodates many more processors than NOGAPS.

#### 2) Comparison with NOGAPS

*n*

^{2}

_{H}

*N*

^{3}

*N*

_{lev}

*N*

^{3}

_{lat}

*N*

_{lev}

*H*and

*T,*as follows:

*H*

^{2}

*N*

_{lev}

*N*

*T*

^{3}

*N*

_{lev}

*T,*for NOGAPS, while it only increases quadratically for SEE-AM,

*H,*provided that

*N*remains fixed.

To understand why the spectral transform method becomes increasingly more expensive than the spectral element method requires revisiting Eqs. (51) and (52). For the spectral element method, the horizontal resolution is governed by *n*_{H} and *N.* Recall that *n*_{H} governs the number of spectral elements, while *N* is the order of the polynomials. This last term is analogous to *N*_{lat} in Eq. (54) for the spectral transform method. To increase the horizontal resolution of a spectral transform model requires increasing *N*_{lat}, which increases the cost by its cube; there is no way around this. In contrast, with the spectral element method one has the choice of increasing either the number of elements or the order of the polynomial. Since the cost increases cubically with *N* and only quadratically with *n*_{H}, then it makes sense to keep the polynomial order fixed and increase the number of elements to obtain higher resolutions. This flexibility is due to the h–p nature of the spectral element method. By keeping *n*_{H} fixed (say *n*_{H} = 1) and increasing *N,* we reach the spectral transform limit of the spectral element method (known as the p-type method). On the other hand, by keeping *N* fixed (say *N* = 1) and increasing *n*_{H}, we obtain the linear finite-element limit of the spectral element method (known as the h-type method). On serial computers the optimal strategy for selecting *n*_{H} and *N* is usually to pick *N* in the range [8, 16] and increase *n*_{H} to yield the desired resolution.

*T,*

*p*version of the spectral element method, then the time step must be decreased quadratically with resolution; however, if we keep

*N*fixed then we can achieve a linear decrease of the time step with resolution that will allow spectral element models to compete with spectral transform models. Let us now compare how the time steps differ for NOGAPS and SEE-AM.

*N*

_{lat}

*N*

_{lat}

*N*= 8. These cost estimates are computed on a per processor basis. Based on these cost estimates, the explicit SEE-AM model will outperform the explicit NOGAPS beyond resolutions of T108, and the semi-implicit NOGAPS beyond T406. Currently, some spectral transform models are running beyond T406, such as the ECMWF model, which uses T511, and NOGAPS is expected to be at or beyond this resolution in the near future. A semi-implicit implementation of SEE-AM will outperform the semi-implicit NOGAPS beyond a resolution of T185, which is well below the current operational resolution of T239. This resolution is obtained by comparing the serial versions of our semi-implicit implementation of SEE-AM, in which we use a conservative estimate of a factor of 2 increase in performance over the explicit SEE-AM; however, this does not necessarily guarantee that we will achieve this gain in the parallel version, but it does provide a good estimate. Nonetheless, much work has been done regarding this issue, and we hope to benefit from the volume of work in the literature on this topic, most notably, the parallel elliptic solvers of Tufo and Fischer (1999) for the Navier–Stokes equations and the work by Loft et al. (2001) and Thomas et al. (2002) for the multilevel shallow water equations.

In Fig. 6 we show a performance comparison on an IBM SP3 for the explicit NOGAPS and SEE-AM models; both models use a resolution of T159 with *N*_{lev} = 24 vertical levels and the explicit SEE-AM time step, Δ*t* = 35 s. The results of this figure are summarized as follows. First, SEE-AM (spectral element) is much faster than NOGAPS (spectral transform) on a per time step and per processor basis. For an equal number of processors, say 150, SEE-AM is more than 2 times faster than NOGAPS. Second, SEE-AM can use many more processors than NOGAPS. At a resolution of T159, NOGAPS can only use 150 processors effectively; note that beyond 150 processors the performance of NOGAPS decreases. In contrast, SEE-AM is able to use 600 processors effectively, and it can accommodate up to 2400 processors. Finally, the results presented in Fig. 6 are very reassuring because they show that SEE-AM scales linearly for increasing processor number. At 600 processors, both the model and the communication network did not suffer any severe penalties. Although we should not expect to get the same type of performance on 2400 processors, it is exciting to anticipate that it may be possible.

The results in Fig. 6 clearly show that SEE-AM is far more efficient than NOGAPS on a per time step basis and per processor basis. However, NOGAPS can use a time step much larger than SEE-AM. In Fig. 7 we plot the results for the semi-implicit NOGAPS, explicit NOGAPS, and explicit SEE-AM for a T159 horizontal resolution with 24 vertical levels. The results shown here are plotted using the maximum time step that each model allows, and they are Δ*t* = 540, 120, and 35 s for the semi-implicit NOGAPS, explicit NOGAPS, and SEE-AM, respectively.

The results of this study are summarized as follows. For small processor numbers NOGAPS outperforms SEE-AM. However, for processor numbers greater than 250 SEE-AM outperforms the explicit NOGAPS. If the linear scalability of SEE-AM were to hold for increasing processor numbers, we would expect SEE-AM to outperform the semi-implicit NOGAPS at around 900 processors. However, if we could double the time step by introducing a semi-implicit implementation, then SEE-AM would outperform the semi-implicit NOGAPS beyond 500 processors. It is possible to further increase the efficiency of SEE-AM because this model has not yet been fully optimized. The results presented in this section should not be taken as the optimal performance of spectral element models but merely as a first attempt at constructing fast and efficient NWP models. We hope to benefit from the work of Thomas et al. (2002) and Loft et al. (2001) on the optimization of spectral element models.

## 6. Results

*L*

_{2}error norms defined as follows:

*q*

_{G}is the computed solution vector,

*q*

_{exact}is the exact solution, and the norm is computed as a broken norm. All the results are computed using 64-bit arithmetic precision.

### a. Barotropic tests

To validate the spectral element discrete operators, we run the model using the shallow water tests 1, 2, and 3 in Williamson et al. (1992). Because these tests admit exact solutions, we are able to plot normalized geopotential, *ϕ,* *L*_{2} error norms. Figure 8 shows that SEE-AM achieves the expected exponential convergence regardless of whether the icosahedral or hexahedral grid is used. Case 1 will not yield exponential convergence because of the nonsmooth nature of the derivatives at the base of the cosine bell.

*N*

_{p}we get

The results for the horizontal operators are summarized as follows. Figures 9 and 10 show that for cases 1 and 2 SEE-AM yields the best accuracy of all the models including the high-accuracy Jakob-Chien et al. (1995) and Taylor et al. (1997) models. For case 1, SEE-AM is almost an order of magnitude more accurate than the Taylor et al. model and is twice as accurate as the Jakob-Chien et al. model. For case 2, SEE-AM is approximately eight orders of magnitude more accurate than these two high-accuracy models. The plateauing of the error for SEE-AM is due to the accuracy-reaching machine precision. Finally, Fig. 11 shows that for case 3 SEE-AM gives better accuracy than all the other models except for the spectral transform model of Jakob-Chien et al., which yields a slightly more accurate result (0.7 × 10^{−9} compared to 1 × 10^{−9}).

### b. Baroclinic tests

Because there are no analytic solutions to the full atmospheric equations, we cannot run test cases as in the barotropic case and compare to exact solutions. Instead, we need to either use test cases in which the outcome is a simple enough pattern that might be easily discerned beforehand or we need to run benchmark test cases run by a vast community. We have chosen to use both types of test cases: the Rossby–Haurwitz wave and the balanced initial state representing the former and the Held–Suarez test and the baroclinic instability testing the latter.

No diffusion operators are included in any of our results for both NOGAPS and SEE-AM. At every time step, the ⅔ triangular truncation is applied to NOGAPS and the elementwise filter is applied to SEE-AM.

#### 1) Rossby–Haurwitz wavenumber 4

*σ*∈ [0, 1], where

*a*is the earth's radius, and

*n*is the wave number. The temperature field is based on the NACA standard atmosphere and is defined as

*T*

*z*

Surface contours of the prognostic variables after a 5-day integration for T159, *N*_{lev} = 24 resolutions of SEE-AM and NOGAPS, are shown in Figs. 12 and 13, respectively. The results between the two models are virtually indistinguishable. This means that both models yield similar values for all of the prognostic variables as well as similar phase speeds—an important property for the successful tracking of tropical cyclones. It should not be surprising that SEE-AM gives identical results to NOGAPS. Both models use the same temporal and vertical discretization methods. The only difference is in the horizontal discretization methods. However, in section 6a, using barotropic test cases, we showed that the spectral element method gives almost identical accuracy to the spectral transform method.

Having established the accuracy of the model, we now turn to the stability of the spectral element model for longer time integrations.

#### 2) Held–Suarez test case

*k*

_{υ},

*k*

_{θ}, and

*θ*

_{eq}]. For this test we use an equivalent resolution to that used in Held and Suarez (1994), namely,

*H*64 (

*n*

_{H}= 8 and

*N*= 8), with 20 vertical levels (

*N*

_{lev}). The mean zonally averaged zonal velocity and temperature are shown as a function of the vertical coordinate

*σ*in Fig. 14. These plots are obtained from a 1200-day integration, with the results sampled every 4 days beginning with day 200. These results compare quite well with those obtained with the spectral transform model in Held and Suarez (1994), where the midlatitude jets in the upper atmosphere are clearly visible (Fig. 14a) and a realistic temperature stratification is maintained (Fig. 14b).

#### 3) Jablonowski–Williamson test cases

The following two cases represent a new set of tests for judging the accuracy and stability of dynamical cores. These tests are introduced in Jablonowski and Williamson (2002).

*p*

_{s}(

*λ,*

*φ*) = 1000 hPa, and the initial wind velocities are defined as

*u*

_{0}= 35 m/s,

*σ*

_{υ}= (

*σ*−

*σ*

_{0})

*π*/2,

*σ*

_{0}= 0.252, and

*σ*is the vertical coordinate. The horizontally averaged temperature is

*T*

_{0}= 288 K, Γ = 0.005 K m

^{−1},

*g*= 9.806 m s

^{−2}, Δ

*T*= 4.8 × 10

^{5}K, and

*σ*

_{t}= 0.2 is the tropopause level. Defining the following functions

*P*is the Exner function and (

*a,*

*ω*) are the radius of the earth and its angular velocity, respectively. The surface geopotential is

*σ*

_{s}= 1. We compare SEE-AM with three well-established models using these test cases:

*Balanced initial state.*For this test case, the atmosphere is initially balanced by the above equations for surface pressure,*p*_{s}; wind velocities, (*u,**υ*); potential temperature,*θ*; and surface geopotential,*ϕ*_{s}. Using these initial conditions, the equations should remain balanced for an indefinite amount of time. Figure 15 shows the normalized surface pressure,*π,**L*_{2}error norm as a function of time for a 30-day period for SEE-AM with H160 horizontal resolution and 24 vertical levels. Note that while the error oscillates with time it remains bounded, which confirms that the initial balanced state is maintained.*Baroclinic instability.*This case is similar to the balanced initial state except that now a perturbation is added to the initial zonal velocity. This perturbation is given bywhereand (*λ*_{c},*φ*_{c}) = (*π*/9, 2*π*/9) and*R*=*a*/10 are the location of the perturbation and its radius.

This perturbation grows until a baroclinic instability develops and then breaks near day 9. Figure 16 shows the minimum surface pressure *p*_{s} as a function of time for SEE-AM against various models including the NCAR spectral transform model (Hack et al. 1992), the National Aeronautics and Space Administration (NASA) Goddard finite-volume model (Yeh et al. 2002), and the German Weather Service icosahedral finite-difference model (Majewski et al. 2002) known as GME; the results of the latter three models are courtesy of C. Jablonowski (2003, personal communication). Figure 17 shows a zoomed in version of Fig. 16. For the gridpoint models, we use the definition of equivalent hexahedral resolution, *H,* which we defined for the barotropic test cases. The results of this case are summarized as follows. Figure 16 shows that all four models are in complete agreement until day 8, at which point the two lower-order models, NASA and GME, diverge from the NCAR and SEE-AM models. NASA and GME compare well with each other throughout the 14-day integration. There are some slight deviations between days 12 and 13 (Fig. 17), but overall they both follow the same pattern. The two higher-order models, NCAR and SEE-AM, compare extremely well with each other throughout the 14-day integration. This can be seen more clearly in Fig. 17, where the pressure curves are directly on top of each other.

## 7. Conclusions

A new dynamical core constructed using the spectral element method based on 3D Cartesian coordinates is presented. The advantages of using Cartesian coordinates are the elimination of the polar singularity; the flexibility to use any grid, including adaptive unstructured grids; and the ease with which the Eulerian model can be converted to a semi-Lagrangian form. The advantage of using the spectral element method is that it achieves the same order of accuracy as the spectral transform method while taking better advantage of distributed-memory computers.

In this paper we show results for icosahedral and hexahedral grids, and in future work we expect to report on the use of adaptive unstructured grids. The exponential accuracy of the spectral element method is illustrated using analytic solutions to barotropic test cases, which confirms that the spectral element method yields at least the same level of accuracy obtained with the spectral transform method. Using baroclinic test cases, we demonstrate that our spectral element atmospheric model gives similar results to spectral transform models, including the U.S. Navy's NWP model and the NCAR climate model. Finally, the performance of the spectral element model is shown to scale linearly with increasing processors—a trait not shared by spectral transform models. Through our comparison of NOGAPS and SEE-AM, we show why the spectral element model will outperform spectral transform models for the types of horizontal resolutions required by future NWP applications.

The results confirm that SEE-AM offers an attractive alternative strategy for constructing future NWP and climate models on parallel computers. In order to make SEE-AM competitive with operational NWP spectral transform models, we are extending the explicit Eulerian model to semi-implicit Eulerian and fully implicit semi-Lagrangian, and we hope to report our findings in the future.

## Acknowledgments

The authors gratefully acknowledge the support of their sponsor, the Office of Naval Research, through Program Element PE-0602435N. We gratefully acknowledge the computing resources from the Department of Defense NAVO Major Shared Resource Center. We would like to thank Christiane Jablonowski for allowing us to use a preprint of her paper and for sharing with us her datasets for the baroclinic instability test case. We would also like to thank Steve Lowder for his assistance in getting SEE-AM running on the NAVO IBM SP3, Jean Côté for fruitful discussions on the construction of atmospheric models using Cartesian coordinates, Todd Ringler for helpful discussions regarding the Held–Suarez test case, and João Teixeira for helpful comments regarding the manuscript.

## REFERENCES

Asselin, R., 1972: Frequency filter for time integrations.

,*Mon. Wea. Rev.***100****,**297–301.Boyd, J. P., 1998: Two comments on filtering (artificial viscosity) for Chebyshev and Legendre spectral and spectral element methods: Preserving boundary conditions and interpretation of the filter as a diffusion.

,*J. Comput. Phys.***143****,**283–288.Cooley, J. W., and J. W. Tukey, 1965: An algorithm for the machine calculation of complex Fourier series.

,*Math. Comput.***19****,**297–301.Côté, J., 1988: A Lagrange multiplier approach for the metric terms of semi-Lagrangian models on the sphere.

,*Quart. J. Roy. Meteor. Soc.***114****,**1347–1352.Dennis, J. M., 2003: Partitioning with space-filling curves on the cubed-sphere.

*Proceedings of the International Parallel and Distributed Processing Symposium,*Nice, France, 6 pp.Foster, I., W. Gropp, and R. Stevens, 1992: The parallel scalability of the spectral transform method.

,*Mon. Wea. Rev.***120****,**835–850.Giraldo, F. X., 2001: A spectral element shallow water model on spherical geodesic grids.

,*Int. J. Numer. Methods Fluids***35****,**869–901.Giraldo, F. X., J. S. Hesthaven, and T. Warburton, 2002: Nodal high-order discontinuous Galerkin methods for the spherical shallow water equations.

,*J. Comput. Phys.***181****,**499–525.Hack, J. J., B. A. Boville, B. P. Briegleb, J. T. Kiehl, P. J. Rasch, and D. L. Williamson, 1992: Description of the NCAR community climate model (CCM2). NCAR Tech. Note NCAR/TN-382+STR, Climate Modeling Section, National Center for Atmospheric Research, Boulder, CO, 108 pp.

Heikes, R., and D. A. Randall, 1995: Numerical integration of the shallow water equations on a twisted icosahedral grid. Part I: Basic design and results of tests.

,*Mon. Wea. Rev.***123****,**1862–1880.Held, I. M., and M. J. Suarez, 1994: A proposal for the intercomparison of the dynamical cores of atmospheric general circulation models.

,*Bull. Amer. Meteor. Soc.***75****,**1825–1830.Hodur, R. M., 1997: The Naval Research Laboratory's Coupled Ocean/Atmosphere Mesoscale Prediction System (COAMPS).

,*Mon. Wea. Rev.***125****,**1414–1430.Hogan, T. F., and T. E. Rosmond, 1991: The description of the Navy Global Operational Prediction System's spectral forecast model.

,*Mon. Wea. Rev.***119****,**1786–1815.Hogan, T. F., M. S. Peng, J. A. Ridout, and W. M. Clune, 2002: A description of the impact of changes to NOGAPS convection parameterization and the increase in resolution to T239L30. NRL Memo. Rep. NRL/MR/7530-02-52, Global Modeling Section, Naval Research Laboratory, Monterey, CA, 16 pp.

Iskandarani, M., D. B. Haidvogel, J. C. Levin, E. Curchitser, and C. A. Edwards, 2002: Multiscale geophysical modeling using the spectral element method.

,*Comput. Sci. Eng.***4****,**42–48.Jablonowski, C., and D. L. Williamson, 2002: Baroclinic instability test with two jets in the midlatitudes.

*Extended Abstracts, 2002 Workshop on the Solution of Partial Differential Equations on the Sphere,*Toronto, ON, Canada, Fields Institute, p. 10.Jakob-Chien, R., J. J. Hack, and D. L. Williamson, 1995: Spectral transform solutions to the shallow water test set.

,*J. Comput. Phys.***119****,**164–187.Karypis, G., and V. Kumar, 1998: Multilevel k-way partitioning scheme for irregular graphs.

,*J. Parallel Distrib. Comput.***48****,**96–129.Komatitsch, D., and J. Tromp, 2002: Spectral-element simulations of global seismic wave propagation—I. Validation.

,*Geophys. J. Int.***149****,**390–412.Loft, R. D., S. J. Thomas, and J. M. Dennis, 2001: Terascale spectral element dynamical core for atmospheric general circulation models.

*Proc. ACM/IEEE SC99 Conf. on High Performance Networking and Computing,*Denver, CO, IEEE Computer Society, CD-ROM, ACM 1-58113-213-X/01/0011.Majewski, D., and Coauthors, 2002: The operational global icosahedral–hexahedral gridpoint model GME: Description and high-resolution tests.

,*Mon. Wea. Rev.***130****,**319–338.Monaco, A. V., and R. T. Williams, 1975: An atmospheric global prediction model using modified Arakawa differencing scheme. NPS Tech. Rep. NPS-51WU75041, Dept. of Meteorology, Naval Postgraduate School, Monterey, CA, 86 pp.

Randall, D. A., T. D. Ringler, and R. P. Heikes, 2002: Climate modeling with spherical geodesic grids.

,*Comput. Sci. Eng.***4****,**32–41.Ringler, T. D., R. P. Heikes, and D. A. Randall, 2000: Modeling the atmospheric general circulation using a spherical geodesic grid: A new class of dynamical cores.

,*Mon. Wea. Rev.***128****,**2471–2490.Ronchi, C., R. Iacono, and P. S. Paolucci, 1996: The “cubed sphere”: A new method for the solution of partial differential equations in spherical geometry.

,*J. Comput. Phys.***124****,**93–114.Rosmond, T. E., 2000: A scalable version of the Navy Operational Global Atmospheric Prediction System spectral forecast model.

,*Sci. Programm.***8****,**31–38.Rosmond, T. E., J. Teixeira, M. Peng, T. Hogan, and R. Pauley, 2002: The Navy Operational Global Atmospheric Prediction System (NOGAPS): Forcing for ocean models.

,*Oceanography***15****,**99–108.Sela, J. G., 1980: Spectral modeling at the National Meteorological Center.

,*Mon. Wea. Rev.***108****,**1279–1292.Simmons, A. J., D. M. Burridge, M. Jarraud, C. Girard, and W. Wergen, 1989: The ECMWF medium-range prediction models development of the numerical formulations and the impact of increased resolution.

,*Meteor. Atmos. Phys.***40****,**28–60.Swarztrauber, P. N., D. L. Williamson, and J. B. Drake, 1997: The Cartesian method for solving partial differential equations in spherical geometry.

,*Dyn. Atmos. Oceans***27****,**679–706.Taylor, M., J. Tribbia, and M. Iskandarani, 1997: The spectral element method for the shallow water equations on the sphere.

,*J. Comput. Phys.***130****,**92–108.Thomas, S. J., R. D. Loft, and J. M. Dennis, 2002: Parallel implementation issues: Global versus local methods.

,*Comput. Sci. Eng.***4****,**26–31.Tomita, H., M. Tsugawa, M. Satoh, and K. Goto, 2001: Shallow water model on a modified icosahedral geodesic grid by using spring dynamics.

,*J. Comput. Phys.***174****,**579–613.Tufo, H. M., and P. F. Fischer, 1999: Terascale spectral element algorithms and implementations.

*Proc. ACM/IEEE SC99 Conf. on High Performance Networking and Computing,*Portland, OR, IEEE Computer Society, CD-ROM.Warburton, T., L. F. Pavarino, and J. S. Hesthaven, 2000: A pseudo-spectral scheme for the incompressible Navier–Stokes equations using unstructured nodal elements.

,*J. Comput. Phys.***164****,**1–21.Williamson, D. L., J. B. Drake, J. J. Hack, R. Jakob, and P. N. Swarztrauber, 1992: A standard test set for numerical approximations to the shallow water equations in spherical geometry.

,*J. Comput. Phys.***102****,**211–224.Yeh, K. S., S. J. Lin, and R. B. Rood, 2002: Applying local discretization methods in the NASA finite-volume general circulation model.

,*Comput. Sci. Eng.***4****,**49–54.

## APPENDIX

### Characteristic Velocity of the Atmospheric Equations

**iˆ**,

**ĵ**,

**k̂**denote the Cartesian directional vectors, and the fluxes are

*F*/∂

*q*are

*U*=

**n**·

**u**, and the maximum wave speed of the atmospheric equations is

**Λ**

_{max}=

*U*+

*ϕ*

The equally spaced sigma coordinate system used in the vertical discretization. The prognostic variables reside at the full levels (solid lines) and the diagnostic variables are at the half levels (dashed lines). No-flux boundary conditions are used at the top and bottom of the atmosphere

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The equally spaced sigma coordinate system used in the vertical discretization. The prognostic variables reside at the full levels (solid lines) and the diagnostic variables are at the half levels (dashed lines). No-flux boundary conditions are used at the top and bottom of the atmosphere

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The equally spaced sigma coordinate system used in the vertical discretization. The prognostic variables reside at the full levels (solid lines) and the diagnostic variables are at the half levels (dashed lines). No-flux boundary conditions are used at the top and bottom of the atmosphere

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The icosahedral grid for (a) *n*_{I} = 2, *N* = 8 and (b) *n*_{I} = 4, *N* = 8

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The icosahedral grid for (a) *n*_{I} = 2, *N* = 8 and (b) *n*_{I} = 4, *N* = 8

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The icosahedral grid for (a) *n*_{I} = 2, *N* = 8 and (b) *n*_{I} = 4, *N* = 8

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The hexahedral grid for (a) *n*_{H} = 4, *N* = 8 and (b) *n*_{H} = 8, *N* = 8

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The hexahedral grid for (a) *n*_{H} = 4, *N* = 8 and (b) *n*_{H} = 8, *N* = 8

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The hexahedral grid for (a) *n*_{H} = 4, *N* = 8 and (b) *n*_{H} = 8, *N* = 8

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The communication stencil required by the elements *P*1, … , *P*9 in processor PROC. The dashed box represents the perimeter values that each processor sends to its neighbors, and *N*1, … , *N*9 represent the elements of the eight neighboring processors (NBR)

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The communication stencil required by the elements *P*1, … , *P*9 in processor PROC. The dashed box represents the perimeter values that each processor sends to its neighbors, and *N*1, … , *N*9 represent the elements of the eight neighboring processors (NBR)

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The communication stencil required by the elements *P*1, … , *P*9 in processor PROC. The dashed box represents the perimeter values that each processor sends to its neighbors, and *N*1, … , *N*9 represent the elements of the eight neighboring processors (NBR)

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The simulation days per wall clock time as a function of processors, *N*_{proc}, for the explicit NOGAPS and SEE-AM, for T159 and *N*_{lev} = 24. Both models use the maximum allowable time step for SEE-AM

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The simulation days per wall clock time as a function of processors, *N*_{proc}, for the explicit NOGAPS and SEE-AM, for T159 and *N*_{lev} = 24. Both models use the maximum allowable time step for SEE-AM

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The simulation days per wall clock time as a function of processors, *N*_{proc}, for the explicit NOGAPS and SEE-AM, for T159 and *N*_{lev} = 24. Both models use the maximum allowable time step for SEE-AM

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The simulation days per wall clock time as a function of processors, *N*_{proc}, for the semi-implicit NOGAPS, explicit NOGAPS, and SEE-AM, for T159 and *N*_{lev} = 24. Each model uses its maximum allowable time step

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The simulation days per wall clock time as a function of processors, *N*_{proc}, for the semi-implicit NOGAPS, explicit NOGAPS, and SEE-AM, for T159 and *N*_{lev} = 24. Each model uses its maximum allowable time step

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

The simulation days per wall clock time as a function of processors, *N*_{proc}, for the semi-implicit NOGAPS, explicit NOGAPS, and SEE-AM, for T159 and *N*_{lev} = 24. Each model uses its maximum allowable time step

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Barotropic cases 1, 2, and 3: The geopotential, *ϕ,* normalized *L*_{2} error as a function of polynomial order, *N,* for the Williamson et al. (1992) shallow water tests 1, 2, and 3 using (a) icosahedral and (b) hexahedral grids. The *n*_{I} = 1 icosahedral grid and *n*_{H} = 1 hexahedral grid are used, and the results are reported for 12, 5, and 5 days for cases 1, 2, and 3, respectively

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Barotropic cases 1, 2, and 3: The geopotential, *ϕ,* normalized *L*_{2} error as a function of polynomial order, *N,* for the Williamson et al. (1992) shallow water tests 1, 2, and 3 using (a) icosahedral and (b) hexahedral grids. The *n*_{I} = 1 icosahedral grid and *n*_{H} = 1 hexahedral grid are used, and the results are reported for 12, 5, and 5 days for cases 1, 2, and 3, respectively

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Barotropic cases 1, 2, and 3: The geopotential, *ϕ,* normalized *L*_{2} error as a function of polynomial order, *N,* for the Williamson et al. (1992) shallow water tests 1, 2, and 3 using (a) icosahedral and (b) hexahedral grids. The *n*_{I} = 1 icosahedral grid and *n*_{H} = 1 hexahedral grid are used, and the results are reported for 12, 5, and 5 days for cases 1, 2, and 3, respectively

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Barotropic case 1: The geopotential, *ϕ,* normalized *L*_{2} error as a function of horizontal resolution, *H,* for the Williamson et al. (1992) shallow water case 1 after 12 days for SEE-AM (thick line), the Jakob-Chien et al. (1995) (spectral transform), the Heikes and Randall (1995) (finite difference), and the Taylor et al. (1997) (spectral element) models. The SEE-AM model uses the *n*_{H} = 1 (*N*_{e} = 6) grid

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Barotropic case 1: The geopotential, *ϕ,* normalized *L*_{2} error as a function of horizontal resolution, *H,* for the Williamson et al. (1992) shallow water case 1 after 12 days for SEE-AM (thick line), the Jakob-Chien et al. (1995) (spectral transform), the Heikes and Randall (1995) (finite difference), and the Taylor et al. (1997) (spectral element) models. The SEE-AM model uses the *n*_{H} = 1 (*N*_{e} = 6) grid

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Barotropic case 1: The geopotential, *ϕ,* normalized *L*_{2} error as a function of horizontal resolution, *H,* for the Williamson et al. (1992) shallow water case 1 after 12 days for SEE-AM (thick line), the Jakob-Chien et al. (1995) (spectral transform), the Heikes and Randall (1995) (finite difference), and the Taylor et al. (1997) (spectral element) models. The SEE-AM model uses the *n*_{H} = 1 (*N*_{e} = 6) grid

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Barotropic case 2: The geopotential, *ϕ,* normalized *L*_{2} error as a function of horizontal resolution, *H,* for the Williamson et al. (1992) shallow water case 2 after 5 days for SEE-AM (thick line), the Jakob-Chien et al. (1995) (spectral transform), the Heikes and Randall (1995) (finite difference), the Taylor et al. (1997) (spectral element), and the Tomita et al. (2001) (finite difference) models. The SEE-AM model uses the *n*_{H} = 1 (*N*_{e} = 6) grid

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Barotropic case 2: The geopotential, *ϕ,* normalized *L*_{2} error as a function of horizontal resolution, *H,* for the Williamson et al. (1992) shallow water case 2 after 5 days for SEE-AM (thick line), the Jakob-Chien et al. (1995) (spectral transform), the Heikes and Randall (1995) (finite difference), the Taylor et al. (1997) (spectral element), and the Tomita et al. (2001) (finite difference) models. The SEE-AM model uses the *n*_{H} = 1 (*N*_{e} = 6) grid

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Barotropic case 2: The geopotential, *ϕ,* normalized *L*_{2} error as a function of horizontal resolution, *H,* for the Williamson et al. (1992) shallow water case 2 after 5 days for SEE-AM (thick line), the Jakob-Chien et al. (1995) (spectral transform), the Heikes and Randall (1995) (finite difference), the Taylor et al. (1997) (spectral element), and the Tomita et al. (2001) (finite difference) models. The SEE-AM model uses the *n*_{H} = 1 (*N*_{e} = 6) grid

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Barotropic case 3: The geopotential, *ϕ,* normalized *L*_{2} error as a function of horizontal resolution, *H,* for the Williamson et al. (1992) shallow water case 3 after 5 days for SEE-AM (thick line), the Jakob-Chien et al. (1995) (spectral transform), the Heikes and Randall (1995) (finite difference), the Taylor et al. (1997) (spectral element), and the Tomita et al. (2001) (finite difference) models. The SEE-AM model uses the *n*_{H} = 1 (*N*_{e} = 6) grid

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Barotropic case 3: The geopotential, *ϕ,* normalized *L*_{2} error as a function of horizontal resolution, *H,* for the Williamson et al. (1992) shallow water case 3 after 5 days for SEE-AM (thick line), the Jakob-Chien et al. (1995) (spectral transform), the Heikes and Randall (1995) (finite difference), the Taylor et al. (1997) (spectral element), and the Tomita et al. (2001) (finite difference) models. The SEE-AM model uses the *n*_{H} = 1 (*N*_{e} = 6) grid

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Barotropic case 3: The geopotential, *ϕ,* normalized *L*_{2} error as a function of horizontal resolution, *H,* for the Williamson et al. (1992) shallow water case 3 after 5 days for SEE-AM (thick line), the Jakob-Chien et al. (1995) (spectral transform), the Heikes and Randall (1995) (finite difference), the Taylor et al. (1997) (spectral element), and the Tomita et al. (2001) (finite difference) models. The SEE-AM model uses the *n*_{H} = 1 (*N*_{e} = 6) grid

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Rossby–Haurwitz wavenumber 4: The surface (a) pressure (hPa), (b) temperature (K), (c) zonal velocity (m s^{−1}); and (d) meridional velocity (m s^{−1}) for SEE-AM, with *H*160 (*n*_{H} = 20, *N* = 8) and *N*_{lev} = 24, for a 5-day integration

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Rossby–Haurwitz wavenumber 4: The surface (a) pressure (hPa), (b) temperature (K), (c) zonal velocity (m s^{−1}); and (d) meridional velocity (m s^{−1}) for SEE-AM, with *H*160 (*n*_{H} = 20, *N* = 8) and *N*_{lev} = 24, for a 5-day integration

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Rossby–Haurwitz wavenumber 4: The surface (a) pressure (hPa), (b) temperature (K), (c) zonal velocity (m s^{−1}); and (d) meridional velocity (m s^{−1}) for SEE-AM, with *H*160 (*n*_{H} = 20, *N* = 8) and *N*_{lev} = 24, for a 5-day integration

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Rossby–Haurwitz wavenumber 4: The surface (a) pressure (hPa), (b) temperature (K), (c) zonal velocity (m s^{−1}), and (d) meridional velocity (m s^{−1}) for NOGAPS, with *T*159 and *N*_{lev} = 24, for a 5-day integration

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Rossby–Haurwitz wavenumber 4: The surface (a) pressure (hPa), (b) temperature (K), (c) zonal velocity (m s^{−1}), and (d) meridional velocity (m s^{−1}) for NOGAPS, with *T*159 and *N*_{lev} = 24, for a 5-day integration

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Rossby–Haurwitz wavenumber 4: The surface (a) pressure (hPa), (b) temperature (K), (c) zonal velocity (m s^{−1}), and (d) meridional velocity (m s^{−1}) for NOGAPS, with *T*159 and *N*_{lev} = 24, for a 5-day integration

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Held–Suarez test: Plots of the (a) mean zonally averaged zonal velocity (m s^{−1}) and (b) mean zonally averaged temperature (K) for SEE-AM after 1200 days for *H*64 (*n*_{H} = 8, *N* = 8) and *N*_{lev} = 20

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Held–Suarez test: Plots of the (a) mean zonally averaged zonal velocity (m s^{−1}) and (b) mean zonally averaged temperature (K) for SEE-AM after 1200 days for *H*64 (*n*_{H} = 8, *N* = 8) and *N*_{lev} = 20

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Held–Suarez test: Plots of the (a) mean zonally averaged zonal velocity (m s^{−1}) and (b) mean zonally averaged temperature (K) for SEE-AM after 1200 days for *H*64 (*n*_{H} = 8, *N* = 8) and *N*_{lev} = 20

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Jablonowski–Williamson balanced initial state: The normalized surface pressure, *π,* *L*_{2} error as a function of days for SEE-AM H160 (*n*_{H} = 20, *N* = 8), with 24 vertical levels

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Jablonowski–Williamson balanced initial state: The normalized surface pressure, *π,* *L*_{2} error as a function of days for SEE-AM H160 (*n*_{H} = 20, *N* = 8), with 24 vertical levels

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Jablonowski–Williamson balanced initial state: The normalized surface pressure, *π,* *L*_{2} error as a function of days for SEE-AM H160 (*n*_{H} = 20, *N* = 8), with 24 vertical levels

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Jablonowski–Williamson baroclinic instability: The minimum surface pressure (hPa) as a function of days for the NASA (finite volume), GME (finite difference), NCAR (spectral transform), and SEE-AM (spectral element, with *n*_{H} = 20 and *N* = 8) models using 26 vertical levels. (The data for the first three models are courtesy of C. Jablonowski)

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Jablonowski–Williamson baroclinic instability: The minimum surface pressure (hPa) as a function of days for the NASA (finite volume), GME (finite difference), NCAR (spectral transform), and SEE-AM (spectral element, with *n*_{H} = 20 and *N* = 8) models using 26 vertical levels. (The data for the first three models are courtesy of C. Jablonowski)

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Jablonowski–Williamson baroclinic instability: The minimum surface pressure (hPa) as a function of days for the NASA (finite volume), GME (finite difference), NCAR (spectral transform), and SEE-AM (spectral element, with *n*_{H} = 20 and *N* = 8) models using 26 vertical levels. (The data for the first three models are courtesy of C. Jablonowski)

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Jablonowski–Williamson baroclinic instability: A zoomed-in view of Fig. 16

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Jablonowski–Williamson baroclinic instability: A zoomed-in view of Fig. 16

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2

Jablonowski–Williamson baroclinic instability: A zoomed-in view of Fig. 16

Citation: Monthly Weather Review 132, 1; 10.1175/1520-0493(2004)132<0133:ASSEEA>2.0.CO;2