## 1. Introduction

A primary objective of the current climate community and its sponsors is to create accurate predictions of future climate on decadal to centennial time scales and a broad spectrum of space scales by improving model-component performance and accuracy, by implementing efficient strategies to coupled model components, and by maximizing throughput on state-of-the-art computers capable of exceptional peak speeds. To assist in this endeavor, we have developed a climate model entitled Spectral Element Atmospheric Climate Model (CAM_SEM). CAM_SEM includes a unique dynamical core, Spectral Element Atmosphere Model (SEAM), coupled at this time to the physics component of the Community Atmosphere Model (CAM) as well as the Community Land Model (CLM), both available as part of the Community Climate System Model (CCSM) effort at the National Center for Atmospheric Research (NCAR), and supported through collaboration with universities and federal agencies [National Science Foundation, Department of Energy (DOE), National Oceanic and Atmospheric Administration, etc.]. These coupled components are currently state of the art, yet are by no means perfect. In this framework the community effort has entrained a large group of global climate modelers and model contributors who currently work to create optimum model components to generate reliable climate predictions, and our effort thereto has focused on the dynamical core (dycore) component. The intensity and scope of this effort supports the observation that notable limitations in the goals presented above still exist, despite the many exciting results that have been achieved to date. Included among the major improvements still needed are a more transparent, efficient, and accurate method of producing *regional climate predictions* involving local mesh refinement, improved numerical quadrature, a more comprehensive technique for predicting the overall global climate, and application of computing methodology that uses the latest in computing hardware most effectively and economically to produce the best prediction/simulation results with the minimal expenditure of resources. Our efforts, described herein, should contribute to improved future predictions.

We note that model climates tend to depart from the true climate to one of their own as integration time progresses, and that increasing model resolution tends to inhibit but not eliminate this process. One signature of climate change may be identified in the frequency and magnitude of deviations from the climate mean, and by the increased frequency of regional events such as hurricanes, or severe storm complexes that have significant impact on relatively short time and space scales. To uncover these phenomena model resolution must be increased, at least locally, with the consequence that processes affecting previously unresolved scales in coarser-grained models must then be considered. Including shorter space scales introduces shorter time scales, thereby further increasing computational requirements. These short time cycles themselves can lead to significant prediction errors, as small computational errors propagate through the system and gradually grow with time (e.g., Lilly 1973). This evolution highlights the importance of understanding in detail the impact of the shorter space scales on predictions and how these scales interact with the larger scales in the coarse-grained portion of the total domain, but also highlights the need to develop model methodology that optimizes the use of computer resources. The methodology we have developed and implemented is formulated to take into account local scaling requirements, because dynamic scalability, or numerical convergence under mesh refinement, is achieved without distorting the overall global prediction. This achievement utilizes the advantageous features of the computers on which the model is run and is ideally suited for prediction on any and all space scales deemed significant, doing so in a seamless and rigorously convergent fashion. To this end we have implemented both explicit and semi-implicit time-stepping methods that are well suited to Reduced Instruction Set Computer (RISC) microprocessors with a deep memory hierarchy (cache) and a hybrid programming model (Fournier et al. 2004; Thomas and Loft 2002).

Our modeling studies have exploited the spectral-element method, first explored within the context of earth-system modeling by the ocean-modeling community (Haidvogel et al. 1997). We developed a pioneering version for the global atmosphere, which we named the Spectral Element Atmosphere Model (Taylor et al. 1997, 1998; Fournier et al. 2000a, b; Thomas et al. 2000, 2001, 2003; Thomas and Loft 2005; St-Cyr and Thomas 2005; Fournier et al. 2004; hereafter FTT). Spectral elements have advantageous properties both for global modeling and for inclusion of regional space scales by using local mesh refinement (LMR), thereby providing higher resolution in regions of strong local variability and generating regional predictions within a global model. The model is self-contained as a *dycore* and can be combined with appropriate physics and other scientific numerical packages to create a global climate model (GCM). We have not recreated the many packages required for a state-of-the-art GCM, but we have coupled SEAM to currently accepted state-of-the art community packages available at NCAR (see above) and have therewith demonstrated the efficacy of SEAM as a suitable and desirable dycore for the CCSM concept.

## 2. Properties of SEAM

Our global dycore, SEAM, offers a number of distinct advantages. It is highly flexible, performing regional and global-scale integrations concurrently in a very straightforward manner. SEAM utilizes the geometric properties of finite-element methods, allows for very convenient LMR and regional detail, is ideally suited to parallel processing by minimizing communication amongst the processors, is very efficient computationally, and has no geometric pole problems. The method for generating this model is well documented and specific details on it can be found in Fournier et al. (2004), including successful experiments with the shallow water equations. Additional integration results with the model can found in Baer et al. (2001) and Wang et al. (2004). We thus provide only a brief summary of the model and its current state in the following. The model originally used surface-relative pressure *σ̃* coordinates in the vertical (Phillips 1957), but this has been changed to hybrid *η* coordinates to more closely follow the NCAR CAM (Collins et al. 2003). In the horizontal dimensions, all selected spherical surfaces in the global atmosphere are on specified *η* levels, on which the flow is to be predicted, and are tiled with an arbitrary number and size of spherical-quadrangular elements. Inscribing a cube inside an associated sphere to represent the earth, and mapping the surface of the sphere to the surface of the cube with a gnomonic projection, generates these elements. Each face is subsequently subdivided arbitrarily as desired, yielding a set of quadrangular elements that cover the surface. The elements can be made uniform if so desired, similar to most GCMs in use today (Fig. 1, lower hemisphere) or can be distributed with high resolution in selected regions (Fig. 1, upper hemisphere). Each element is itself subdivided into a 2D *N* ×* N* Gauss–Lobatto grid; experiments with the model indicate that an 8 × 8 array (i.e., 2D polynomials of degree *N* ≤ 7) is optimal in minimizing computational errors with maximum computational efficiency (Taylor et al. 1997). Using finite-element methodology (Cullen 1979), a set of *N* interpolating basis functions is selected and applied to the grid points in the array in both dimensions, and the same *N* ^{2} functions are used in the maps of all elements to the standard element [−1, 1]^{2}. The dependent variables appropriate to the model equations are expanded in these basis functions with time-dependent coefficients. Each basis function vanishes at every node excepting its cardinal one. The global test functions we chose are conjunctions of the elementary basis functions. To set up the format for numerical computation, the model equations in the basis-function expansion are multiplied by a test function and integrated over the spherical surface; this is done at each point over the entire global domain. Careful consideration is given to the boundaries where the elements meet and continuity of functions between elements is preserved. The resulting equations for the time-dependent basis-function coefficients are unique and define the tendencies of the dependent variables at each node within an element and for all elements. The resulting equations in computational form are represented on a time grid and can be advanced in time by various methods.

The basis functions we used are Legendre cardinal functions (for an example, see Fig. 3 of Fournier et al. 2004) and are the same functions in both dimensions over the grid. Moreover, these functions enable Gauss–Lobatto quadrature, because the grid in each element uses mappings of the nodes of this quadrature. Based on the quadrature, the integral equations are reduced to a set of summations over the quadrature points, creating a finite number of numerical equations for the unknown time-dependent coefficients. These choices result in an extremely simple system with a diagonal mass matrix.

In addition to the conversion to *η* coordinates, we have implemented a semi-Lagrangian transport scheme for moisture variables divided into horizontal and vertical transport. The horizontal transport on the spectral-element grids uses spectral-element interpolation with a maximum/minimum limiter imposed. The vertical transport in *η* coordinates uses Hermite cubic interpolation and requires that a sufficient condition for monotonicity be satisfied.

We have noted that regional climatic events may have a strong impact on global climate, and that a process for predicting at higher resolution locally over those domains where important smaller-scale events occur could be fundamental to a successful prediction. To this end we have implemented the “picture framing” method as a procedure for LMR. It is a simple approach to resizing the local grid and can be applied anywhere over the globe and at any time if desired. Moreover it is exceptionally easy to implement with the spectral-element grid. It is comparable to the stretched-grid approach of Fox-Rabinovitz et al. (2001), although it is somewhat more flexible. As Fox-Rabinovitz et al. point out, if the grid reduction is done too aggressively, computational dispersion may develop and ultimately compromise the accuracy of the prediction. We have tested our use of picture framing in some detail using the shallow water equations version of SEAM and experienced no ill effects (see Fournier et al. 2004). Furthermore, our use of picture framing in a full climate model, discussed later in this paper, also disclosed no negative effects anywhere in the model domain, and most notably in the domain immediately surrounding its application. Despite this apparent success, we are fully aware of this potential problem and are actively pursuing alternative methods to grid reduction, and we discuss this further in the conclusions.

Our current implementation of picture framing allows a reduction of grid length by a factor of 3 in passing from the uniform grid through one additional layer of partially reduced grids, and the process may be repeated if desired or needed. Moreover the process may also be applied to multiple regions. A demonstration of how this procedure works is indicated in Fig. 2 using a 4 × 4 coarse grid of elements as an example. Starting from the coarse outer grid, one passes through corner elements containing three interior subelements. On the sides one passes through elements with seven interior subelements. Finally, on the interior of this ring of corner and side elements, one has the expanded grid of coarse elements subdivided into nine subelements. This represents the 3 × 3 reduction in element size, and the LMR domain defined in this way may be any desired size. Indeed, the process may be repeated inside of the reduced region. Additional details may be found in Fournier et al. (2004). For some of the model integrations with LMR to be discussed subsequently, we have utilized picture framing over the continental United States. Figure 3 shows the elements for these runs wherein each element contains an 8 × 8 grid of points.

Most exciting about spectral-element model formulation is the ease with which the calculations are performed on a parallel processor and the excellent parallel efficiency and scalability of the method. Indeed the spectral-element method is ideal for symmetric multiprocessing (SMP) clusters. The spectral elements are distributed uniformly among the processors. Communication costs are minimal because each element needs only to communicate its boundary data with neighboring elements; hence each processor needs only to communicate its elements’ boundary data with neighboring processors. The dense matrix-vector multiplications that must be performed within each element are localized to each processor and require no outside communication. For a wide variety of problem and machine sizes, we have found that the parallel processor-doubling efficiency of the algorithm never drops below 99%. Furthermore, the method is well suited for the latest generation of supercomputers utilizing cache-based microprocessors. These computers work best with well-“blocked” codes, where many computations are performed on small blocks of data that fit into the cache of the processor. The elements in the spectral-element method provide a natural way to block the code. Unlike spherical-harmonic spectral methods, the size of these blocks in the spectral-element method is small and independent of resolution. We have achieved good performance at those resolutions where global spectral models start to experience cache “thrashing.”

It has been our observation that the spectral-element method appears to be a most efficient and natural way to achieve a high-order, spectrally accurate finite-element discretization, and in particular, to simplify regional modeling. The following examples of experiments with the method should demonstrate the basis for our confidence. They include running SEAM as a 3D dycore with simple physics, and gradually building the model to include full physics and boundary forcing by coupling SEAM to state-of-the-art forcing algorithms.

## 3. Experimental results with SEAM as a 3D dycore

Based on our runs with SEAM in the shallow water mode (Taylor et al. 1997; Fournier et al. 2004) using the test suite provided to the community by Williamson et al. (1992), we have been able to establish the model’s flexibility, its ability to produce regional detail under LMR, its accuracy and computational efficiency when compared with other similar models, and its advantages when using parallel processors. We then ran SEAM under various representations as a 3D dycore and have found that in all circumstances it performed well, better than or equivalently to competitive models. We first determined this by testing SEAM with simple zonal forcing as suggested by Held and Suarez (1994, hereafter HS) using 384 elements distributed uniformly over the globe with an 8 × 8 grid in each. This is roughly comparable to T85 truncation in a spectral model. As with HS, we used 20 equally spaced *σ* levels. The initial conditions for SEAM were identical to those used by HS. Comparisons of our model output variables (SEAM) with those given by HS for both a spectral model (T63) and a finite-difference gridpoint model (144 points on a latitude circle) from the final 1000-day time averages of 1200-day runs show that all three models produce essentially the same results. Additional details of this experiment, including computational features, may be found in FTT. In subsequent studies with SEAM in this model configuration (dycore with HS forcing) we focused on the model’s ability to handle variable mesh sizes, and what the impact of these variations was on the model’s predicted output. In addition, we investigated, with this version, SEAM’s ability to integrate in the LMR mode. The results of these investigations were used as a baseline for the fully forced CAM_SEM climate version of the model.

### a. Resolution and topographic effects

Using the 3D dycore version of SEAM with HS forcing, we studied the effects of varying resolution with and without the inclusion of the earth’s topography. Starting from a state of rest, we integrated the model for 1000 days, thereby allowing the model to come to equilibrium. We then averaged the model output data over the last 350 days and analyzed those results. We interpolated all data to pressure surfaces from *σ* surfaces and to a regular latitude–longitude grid after time averaging. We examined a number of variables including the temperature variance, meridional wind variance, temperature and zonal wind, averaging the model output over time and longitude. We also averaged over height and time, thereby allowing a variety of perspectives on the data. In our experiments we considered three different resolutions: coarse mesh (∼T30), medium mesh (∼T42), and fine mesh (∼T85). The following four experiments were run: (a) coarse mesh without topography, (b) coarse mesh with topography, (c) medium mesh with topography, and (d) fine mesh with topography. The results of the experiments show features that are expected from the introduction of topography and changes in resolution. All the variables we assessed were antisymmetric in latitude when topography was introduced, independent of model resolution. This is clearly anticipated because topographic forcing is not uniform. As resolution is increased, we noted some differences amongst the variables. Whereas the temperature field is very robust and does not show much change, there is a distinct increase in the temperature-variance maxima with increased resolution, and the definition of these maxima becomes more precise as resolution increases. All predicted features are consistent with the introduction of topography and increasing resolution, and support our confidence that SEAM is performing accurately under the limitations of idealized forcing.

### b. LMR and companion studies

Based on the results of Fox-Rabinovitz (2000) and Fox-Rabinovitz et al. (1997, 2000, 2001), whose stretched-grid GCM has successfully produced variable-resolution global predictions, we have made predictions with the SEAM dycore and simple HS physics applying global and regional scaling concurrently, to check SEAM’s skill using LMR under conditions similar to Fox–Rabinovitz et al. (2001, hereafter F-R). At present we have used only the continental United States as the subdomain for LMR as did F-R (Fig. 3). In our experiments, each element in this regional domain was subdivided into nine elements using picture framing; note that each smaller element also contains the 8 × 8 quadrature grid. Except for inclusion of the LMR region, the model was identical to the one used for the topographic experiments discussed in the previous section (3a). The runs were for identical conditions given there, as was the analysis. In this case we made only two runs using the coarse mesh, one without topography *but with* LMR and the other with both topography *and* LMR. The topography used for this experiment was taken from the T42 topography archives available at NCAR and interpolated to the SEAM grid. The primary purpose was to establish the applicability of LMR in the 3D version of the SEAM (dycore); we had previously established its effectiveness with the shallow water equations (Fournier et al. 2004, section 3d). Analysis of the integration results indicates that LMR is working successfully. We noted from both the temperature variance field and the meridional wind variance field that even without topography, the maximum in the Northern Hemisphere is enhanced, indicating that higher resolution brings out details that are missed without LMR. This effect is further enhanced when topography is included, as would be expected. Additionally, we observed a strengthening in the amplitude of the variance of the meridional wind velocity in planetary wavenumbers 3–6 with the inclusion of LMR, even without topography, and a spreading of this effect to more waves when topography is included. These results strongly suggest that LMR is working successfully in the dycore, and we thus proceeded to investigate a more realistically forced model. Analysis of that experiment, to be presented, will support and elaborate on the results reported in this section.

## 4. CAM_SEM: Complete climate model with SEAM as dycore

### a. Development issues

To introduce physics into a climate model requires the availability of a physics package for coupling with a dycore. We have elected to use the state-of-the-art physics package developed for the CCSM system at NCAR (CAM), and we have coupled this algorithm, as well as the CLM algorithm, to SEAM. Additionally we have utilized the coupler developed for use in the CCSM system. That coupler takes the tendencies of the variables calculated in the individual component models and combines them, extrapolates them to the next time level, and sends them back to their respective component models for the calculation of new tendencies. Thus the tendencies for the variables describing dynamic processes come from the dycore (in our case SEAM), the tendencies for the variables describing the physical processes such as radiative heating and convection come from CAM, and the tendencies for land surface process variables are calculated in CLM. The cubed-sphere grid geometry atmospheric dycore (SEAM) that we use for climate prediction has an unstructured grid, and adaptation to the structured grid component models (CAM and CLM) was needed when coupling them to SEAM. We consequently modified the NCAR coupler to fit the CAM physics to the SEAM grid. Additionally, and particularly important, we adapted the NCAR CLM module, wherein surface vegetation types cannot be straightforwardly interpolated, and we have succeeded in accurately producing the surface data fields on the SEAM grid. Our modifications to the coupler have been incorporated through programmable switches that broaden the options available to our research community, since they now allow other unstructured grids to be coupled to the CAM physics and CLM packages. Because there are a number of projects under way that use unstructured grids, this availability should save the research community substantial repetitious reprogramming, and we anticipate that the modified coupler will get broad distribution.

As a corollary to our coupling effort, we have implemented a semi-Lagrangian transport scheme for moisture advection in SEAM. We divide the transport into horizontal and vertical. The horizontal transport is on spectral-element nodes and we use the second-order Runge–Kutta scheme (RK2) for trajectory calculations, with the local spectral-element (Legendre) base for interpolation. Finally, we impose a maximum/minimum limiter on scalar constituents. The vertical transport is in *η* coordinates, similar to CAM_EUL (Collins et al. 2003), using RK2 for trajectory calculations, Hermitian cubic interpolation, and imposing a sufficient condition for monotonicity. We have also implemented RK4; this is a one-step scheme, consistent with the semi-Lagrangian scheme for the spectral-element method (Deville et al. 2002). It also uses subcycling for dynamics including four Δ*t*/4 steps for dynamics and one Δ*t* step for physics. All these modifications were made to simplify the matching characteristics of SEAM in communication with the full physics module.

### b. Experiments with the CAM physics package and SEAM

For experiments with the physics coupled model (CAM_SEM), the initial and boundary data were selected using a “distance weighted” method to interpolate CAM T42 data to SEAM grids. When the CLM is included, we used a distance-weighted method to interpolate the “raw” data to SEAM grids to establish the surface boundary data. We have done a variety of experiments with CAM_SEM as detailed below. Tests to date indicate that the model is running in a stable fashion, and producing results that are similar to CAM_ EUL (the NCAR Eulerian dynamical core coupled to CAM physics). That model is thoroughly tested, and comparison with it should give a good indication of the reliability and stability of CAM_SEM. Because CAM_ EUL was routinely run with T42 truncation, all the experiments described in the sequel have been truncated as closely as possible in spectral elements to T42. It should also be noted that in all experiments with SEAM as the dycore, explicit time integration is used.

#### 1) Held–Suarez experiment

These experiments are similar to those reported in section 3 above, but evolve from CAM_SEM rather than from SEAM alone. Thus we used here the CAM_ SEM with SEAM dynamical core (*η* version) and CAM physics but simplified to HS physics. Additionally, the runs began with climatological initial conditions taken from NCAR archives. The surface boundary conditions were taken from simplifications to CLM (also coupled to CAM_SEM) either with or without topography, and in one experiment the surface was given as an aquaplanet. For each of these experiments we have run CAM_EUL with identical conditions. Hence comparisons of CAM_SEM output with that of CAM_EUL immediately identify model-induced differences. Because CAM_EUL is a carefully analyzed model, we should be able to clearly assess the capabilities of CAM_SEM. Pilot runs of 10-days duration were first made for the various conditions stated above to assure that CAM_SEM did not start off on its own climate, and comparisons of output with that from CAM_EUL showed that the results of the two models were virtually identical. With that assurance, a longer integration was undertaken. Both models were then run for 1200 days with Δt = 450 s, again with HS forcing and the real initial conditions as noted above, but without topography. Instantaneous model values of relevant variables were recorded at 5-day intervals during the integrations and interpolated to a common latitude–longitude grid. These data were then time averaged from day 450 to 1200. Analysis of the results from both models showed that they were producing almost identical predictions, particularly for the temperature and zonal wind fields. As an example, Fig. 4 shows the temperature variance and meridional wind variance for both models on a latitude–pressure diagram (longitudinal values were averaged for this depiction) and except for minor differences in the Southern Hemisphere, the results are very similar.

#### 2) Full physics experiment

Based on our results with CAM_SEM using the simple forcing of HS in the CAM physics module [see section 4b(1)], we proceeded to run the model with the full CAM physics and CLM coupled modules. The initial conditions applied were selected to be identical to those used for an archived run of CAM_EUL as were the boundary conditions that were provided by the CLM. The integration continued for 1200 days with Δ*t* = 1200 s. Instantaneous model values of relevant variables were recorded at 5-day intervals during the integrations and interpolated to the CAM_EUL latitude–longitude grid. These data were then time averaged from day 450 to 1200. We compared the integration results with both CAM2.0.1 (the NCAR-archived CAM_EUL run with identical conditions) and NCEP reanalysis data that were available for the same period. To demonstrate that CAM_SEM is integrating properly for the time period indicated, Fig. 5 shows the average annual zonal wind and temperature fields on a latitude–height diagram, including CAM_SEM, CAM_ EUL and the corresponding NCEP reanalyses. No significant differences amongst these diagrams are evident, suggesting that CAM_SEM is performing satisfactorily. Plots of additional variables lead to a similar conclusion.

#### 3) Local mesh refinement experiments

Analogous to the experiments in the HS framework utilizing LMR and discussed in section 3b, we undertook simulation experiments with CAM_SEM (including full CAM physics and CLM) incorporating both global and regional scaling, and applied modeling conditions similar to F-R, which included one subdomain for LMR—the continental United States. That region had its grid expanded by a factor of 3 in each horizontal dimension using picture framing and is described in Fig. 3. Except for the inclusion of this LMR region, the global model domain, forcing, and initial conditions were identical to the ones used in the experiments described above in sections 4b(1) and 4b(2), including topography. An identical companion run but without the LMR region, that is, coarse resolution globally, was made for comparison purposes. These integrations covered 1200 days with Δ*t* = 600 s. Instantaneous model values of relevant variables were recorded at 5-day intervals during the integrations, interpolated to the CAM_EUL latitude–longitude grid, and time averaged from day 600 to 1200. Integration results from the runs with and without LMR were then compared. The results indicated that inclusion of a high-resolution domain as applied in this investigation does not significantly affect the global distribution of variables. This is supported by Fig. 6, which shows the strong similarity of the average annual zonal wind and temperature fields on a latitude–height diagram for CAM_SEM of both the run with LMR (denoted LMR_USA) and the run with the uniform grid. The same result was seen for other variables including the temperature variance and meridional wind variance. Horizontal maps of the predicted variables also support the observation that inclusion of LMR in the model does not significantly influence the global prediction results, but provides more detail in the LMR region. Figure 7 depicts the time-averaged sea level pressure for both integrations considered herein, and shows only small increased values in the tropical Pacific and mid–Northern Hemisphere for the case with LMR. However, one can see significantly more detail in the averaged precipitation rate over the LMR region from the LMR_USA integration by comparing the regional maps for the two cases on Fig. 8. These results suggest that our model incorporating LMR is working satisfactorily and encouraged us to experiment further with longer runs.

#### 4) An AMIP experiment

The Atmospheric Model Intercomparison Project (AMIP) was established to identify the systematic errors of atmospheric climate models run with realistic initial and boundary conditions (see Gates 1992). Many international modelers contributed results of their integrations to this project and a large body of data thereby became available for intercomparison. These data are stored and made available in standard format by the Program for Climate Model Diagnosis and Intercomparison at the Lawrence Livermore National Laboratory, DOE. It seemed reasonable at this point in the development of CAM_SEM to subject it to the conditions proposed by AMIP to determine if its prediction capability falls within the range of climate models that have contributed to AMIP. A version of CAM_EUL that contained all the conditions required for submission to the AMIP has been run at NCAR, and the appropriate model output has been submitted to AMIP. The same output has been archived at NCAR and denoted CAM2. Because CAM2 represents a community effort and is clearly one of the most advanced models to contribute to AMIP, and because we have used CAM forcing in CAM_SEM, we chose to run CAM_SEM with identical initial and boundary conditions as CAM2 and with its grid set as near as possible to T42, as our version of an AMIP experiment. Our integration covered the period 1979–1998 for which input data were available. Following the integration, we compared CAM_SEM output with the archives of CAM2. Subsequent to the CAM2 run, the NCAR group modified their coupled forcing algorithm and reran CAM2 with this new package as an additional contribution to AMIP, and that output, denoted CAM3, is also available as an archive. Finally, the NCAR archives also contain some observational data taken from reanalysis data that we used for comparison with our model output.

Figure 9 describes the annual surface temperature as a function of latitude, comparing CAM_SEM with CAM2 in Fig. 9a and with NCEP reanalysis data in Fig. 9b. To show how CAM_SEM performs for this variable when compared with the prediction from a model with more advanced physics parameterization, we include the results from CAM3 in Fig. 9b. All four curves have a very similar pattern with only small fluctuations near the South Pole. The surface pressure plots also showed similarly close correspondence amongst these variables (not shown). More interesting is the distribution of the predicted annual total cloud cover presented in percent as a function of latitude on Fig. 10 for CAM_SEM, CAM2, NCEP reanalysis data, and again for reference we include CAM3. Note that CAM_SEM distributes somewhat more like CAM3 in the midlatitudes of both hemispheres but differs from both CAM2 and CAM3 as one approaches the South Pole. By comparison to the reanalysis data curve, all three models differ substantially from it and are more like each other. This is undoubtedly a result of inaccuracies in the forcing algorithm, which is not yet capable of fully describing the true state of the atmosphere, even with its current sophistication. Nevertheless, it suggests that CAM_SEM is in the same ballpark with CAM2 and with CAM3, despite the differences in the physics of the latter.

The annual vertical distributions of temperature and zonal wind with latitude for CAM_SEM and CAM2 are shown in Fig. 11, from which we note that the two models appear quite similar in the structure of the variables, with CAM_SEM a bit cooler in the polar lower stratosphere. These distributions are also very similar to those of Fig. 5, which describes the full physics experiment discussed in section 4b(2). Clearly the average global characteristics of the models are reproducible. With reference to variable distributions on horizontal surfaces, we present the annual surface pressure maps of CAM2 and CAM_SEM in Fig. 12. Note the remarkable similarity in output of the two models. Other maps of different variables on various surfaces also show strong similarity. However, some variables are difficult to predict, as noted by the total cloud cover shown in Fig. 10. The annual distribution of this variable on a surface map for CAM_SEM and CAM2 is presented in Fig. 13. For reference, an observational data analysis (Warren et al. 1986, 1988) is also included. To demonstrate the impact of model forcing relative to the impact of the dycore, we include the results from CAM3. For this variable we note substantial differences. Again we see that all three models compare more favorably with each other than with the distribution given from observational data. However CAM3 is closer to the Warren et al. data than the other two, signifying that the changes in the physics algorithm have made distinct improvements. CAM_SEM and CAM2 are more alike, as should be expected. This again substantiates our contention that CAM_SEM is running successfully.

## 5. Conclusions and plans

We have developed an alternate climate model dycore denoted SEAM in an effort to make climate modeling more flexible, faster, and to allow for seamless incorporation of LMR. The latter enables the prediction of climate on many relevant scales concurrently, as well as enhancing the overall prediction based on the impact of nonlinear interactions of all incorporated scales. The model is founded on the application of spectral elements, which have the advantage of an unstructured grid and allow for easy application of LMR. We generated a comprehensive atmospheric climate model by coupling SEAM to state-of-the-art forcing algorithms (CAM_SEM). Further coupling to models that describe the oceans and cryosphere will produce a comprehensive earth-systems model.

The detailed properties of SEAM have been enumerated in the foregoing text and suitably referenced, and the model itself has been tested systematically. Starting with a thorough assessment using the shallow water equations, the model was subjected to detailed comparison studies as a stand-alone dycore with simple physics to a complete atmospheric climate model incorporating currently best understood physics. It was evaluated using various truncations with or without LMR. Its performance on careful comparison with a modern state-of-the-art atmospheric climate model—NCAR’s CAM2—indicates that CAM_SEM is predicting satisfactorily at this point and is worthy of further development.

One reason we use the spectral-element concept is that it is inherently more efficient in terms of computer costs than most presently used numerical methods. However, there is a resurgence of interest in this problem, and efforts to apply other methodologies such as finite volume, discontinuous Galerkin, and others may ultimately show additional significant computing efficiencies. Our experiments with the dycore indicate that on integration with parallel processing computers the model has definite speedup benefits when compared with conventional models because its computational structure takes advantage of the computer’s design. To optimize this feature for the entire model, the components of our coupled system other than the dycore may need programming redesign to take full advantage of the computing efficiencies noted. Hopefully the forcing algorithms will achieve the computational benefits found for the dycore. This question is under current investigation. One promising development in this area is the application of neural networks, which has been shown to give dramatic speedup when applied to the radiation algorithm (Krasnopolsky et al. 2005).

Having now created a working GCM that incorporates state-of-the-art packages, we are directing our efforts to make the model yet more efficient and accurate. Much of this initiative relates to the seamless regional prediction capability of the model. Mesh generation is fundamental to the success of this process, and we have had satisfactory results using the picture framing method. However, there are other more recently developed tools now available that could improve the accuracy and calculation speed of our predictions and we will test suitable ones. We are currently exploring the efficacy of using meshless collocation in the spectral elements comprising the LMR domain, a procedure that allows for great freedom in the rate at which grid reduction takes place.

With reference to LMR, a process that we deemed essential for our model and which subsequently factored significantly into our selection of spectral elements as the computational methodology for CAM-SEM, the physics forcing package that we now couple to our model is based on parameterizations designed for global-scale models and needs testing for its suitability in our model’s LMR regions. As possible alternatives if needed, existing parameterization schemes used in high-resolution (i.e., mesoscale) models such as the Weather Research and Forecasting model, the fifth-generation Pennsylvania State University–NCAR Mesoscale Model, and superparameterization, a new parameterization scheme currently under development (see Randall et al. 2003), may be more accurate and will be assessed and tested if appropriate. If self-scaling parameterizations exist and are available, we will also test them in CAM_SEM.

## Acknowledgments

The research described herein was supported by Grant DEFG0201ER63247 to the University of Maryland from the DOE/BER/CCPP program. Computing resources were provided by DOE labs at NERSC and ORNL. Additional computing resources were provided by NCAR/SCD. Thanks also go to Mark Taylor for his efforts in applying spectral elements to atmospheric models.

## REFERENCES

Baer, F., J. J. Tribbia, and M. Taylor, 2001: Global and regional atmospheric modeling using spectral elements.

*IUTAM Symposium on Advances in Mathematical Modeling of Atmosphere and Ocean Dynamics,*P. F. Hodnett, Ed., Kluwer Academic, 81–86.Collins, W. D., and Coauthors, cited. 2003: Description of the NCAR Community Atmosphere Model (CAM2). [Available online at http://www.ccsm.ucar.edu/models/atm-cam/docs/description/.].

Cullen, M. J. P., 1979: The finite element method.

*Numerical Methods Used in Atmospheric Models,*Vol. II, A. Kasahara, Ed., WMO GARP Publication 17, 302–337.Deville, M. O., P. F. Fischer, and E. H. Mund, 2002:

*High-Order Methods for Incompressible Fluid Flow. Cambridge Monogr. on Applied and Computational Mathematics,*No. 9, Cambridge University Press, 528 pp.Fournier, A., M. A. Taylor, and J. Tribbia, 2000a: Spectral element method. Part 1: Numerical algorithms.

*Proc. Eighth Annual Conf. of the CFD Society,*Montreal, QC, Canada, CFD Society of Canada, 173–180. [Available online at ftp://ftp.cgd.ucar.edu/pub/fournier/publications/AlNu.ps.gz.].Fournier, A., M. A. Taylor, L. M. Polvani, and R. Saravanan, 2000b: Spectral element method. Part 2: Numerical simulations.

*Proc. Eighth Annual Conf. of the CFD Society,*Montreal, QC, Canada, CFD Society of Canada, 181–188. [Available online at ftp://ftp.cgd.ucar.edu/pub/fournier/publications/NuSi.ps.gz.].Fournier, A., M. A. Taylor, and J. J. Tribbia, 2004: The Spectral Element Atmospheric Model: High-resolution parallel computation and response to regional forcing.

,*Mon. Wea. Rev.***132****,**726–748.Fox-Rabinovitz, M. S., 2000: Regional climate simulation of the anomalous U.S. summer events using a variable-resolution stretched-grid GCM.

,*J. Geophys. Res.***105****,**D24. 29635–29646.Fox-Rabinovitz, M. S., G. L. Stenchikov, M. J. Suarez, and L. L. Takacs, 1997: A finite-difference GCM dynamical core with a variable resolution stretched grid.

,*Mon. Wea. Rev.***125****,**2943–2968.Fox-Rabinovitz, M. S., G. L. Stenchikov, M. J. Suarez, L. L. Takacs, and R. C. Govindaraju, 2000: A uniform and variable resolution GCM dynamical core with realistic orography.

,*Mon. Wea. Rev.***128****,**1883–1898.Fox-Rabinovitz, M. S., L. L. Takacs, M. J. Suarez, and R. C. Govindaraju, 2001: A variable resolution stretched grid GCM: Regional climate simulation.

,*Mon. Wea. Rev.***129****,**453–469.Gates, W. L., 1992: AMIP: The Atmospheric Model Intercomparison Project.

,*Bull. Amer. Meteor. Soc***73****,**1962–1970.Haidvogel, D. B., E. N. Curchitser, M. Iskandarani, R. Hughes, and M. Taylor, 1997: Global modeling of the ocean and atmosphere using the spectral element method.

,*Ocean–Atmos.***35****,**505–531.Held, I., and M. Suarez, 1994: A proposal for the intercomparison of the dynamical cores of atmospheric general circulation models.

,*Bull. Amer. Meteor. Soc.***75****,**1825–1830.Krasnopolsky, V., M. S. Fox-Rabinovitz, and D. Chalikov, 2005: New approach to calculation of atmospheric model physics: Accurate and fast neural network emulation of long wave radiation in a climate model.

,*Mon. Wea. Rev.***133****,**1370–1383.Lilly, D. K., 1973: Lectures in sub-synoptic scales of motions and two-dimensional turbulence.

*Dynamic Meteorology,*P. Morel, Ed., D. Reidel, 353–418.Phillips, N. A., 1957: A coordinate system having some special advantages for numerical forecasting.

,*J. Atmos. Sci.***14****,**184–185.Randall, D., M. Khairoutdinov, A. Arakawa, and W. Grabowski, 2003: Breaking the cloud-parameterization deadlock.

,*Bull. Amer. Meteor. Soc.***84****,**1547–1564.St-Cyr, A., and S. J. Thomas, 2005: Nonlinear operator integration factor splitting for the shallow water equations.

,*App. Numer. Math.***52****,**4. 429–448.Taylor, M., J. J. Tribbia, and M. Iskandarani, 1997: The spectral element method for the shallow water equations on the sphere.

,*J. Comput. Phys.***130****,**92–108.Taylor, M., R. Loft, and J. Tribbia, 1998: Performance of a spectral element atmospheric model (SEAM) on the HP Exemplar SPP2000. NCAR Tech. Rep. TN-439+EDD, 16 pp.

Thomas, S. J., and R. D. Loft, 2002: Semi-implicit spectral element atmospheric model.

,*J. Sci. Comput.***17****,**339–350.Thomas, S. J., and R. D. Loft, 2005: The NCAR spectral element climate dynamical core: Semi-implicit, Eulerian formulation.

,*J. Sci. Comput.***25****,**307–322.Thomas, S. J., R. D. Loft, W. Spotz, and A. Fournier, 2000: Semi-implicit scheme for the spectral element atmospheric model.

*Proc. Eighth Annual Conf. of the CFD Society,*Montreal, QC, Canada, CFD Society of Canada, 231–238.Thomas, S. J., R. D. Loft, A. Fournier, and J. Tribbia, 2001: Parallel spectral element dynamical core for atmospheric general circulation models.

*Proc. Ninth Annual Conf. of the CFD Society,*Waterloo, ON, Canada, CFD Society of Canada, 69–74.Thomas, S. J., J. M. Dennis, H. M. Tufo, and P. Fischer, 2003: A Schwarz preconditioner for the cubed-sphere.

,*SIAM J. Sci. Comput.***25****,**442–453.Wang, H., J. Tribbia, F. Baer, A. Fournier, and M. Taylor, cited. 2004: Recent development and applications of a spectral element dynamical core. [Available online at http://www. ccsm.ucar.edu/working_groups/Atmosphere/Presentations/20040309/04.pdf.].

Warren, S. G., C. J. Hahn, J. London, R. M. Chervin, and R. L. Jenne, 1986: Global distribution of total cloud cover and cloud type amounts over land. NCAR Tech. Note TN-273+STR, 29 pp. + 199 maps.

Warren, S. G., C. J. Hahn, J. London, R. M. Chervin, and R. L. Jenne, 1988: Global distribution of total cloud cover and cloud type amounts over the ocean. NCAR Tech. Note TN-317+STR, 42 pp. + 170 maps.

Williamson, D., and Coauthors, 1992: A standard test set for numerical approximations to the shallow water equations in spherical geometry.

,*J. Comput. Phys.***102****,**211–224.

Example of picture framing, subdividing each of the 2 × 2interior elements of the original 4 × 4 set into 3 × 3 elements.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Example of picture framing, subdividing each of the 2 × 2interior elements of the original 4 × 4 set into 3 × 3 elements.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Example of picture framing, subdividing each of the 2 × 2interior elements of the original 4 × 4 set into 3 × 3 elements.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Element mapping, using LMR with picture framing over the continental United States. Note that each element contains an 8 × 8 grid (not shown).

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Element mapping, using LMR with picture framing over the continental United States. Note that each element contains an 8 × 8 grid (not shown).

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Element mapping, using LMR with picture framing over the continental United States. Note that each element contains an 8 × 8 grid (not shown).

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Distributions of (left) *temperature variance* and (right) *meridional wind variance* from both (top) CAM_SEM and (bottom) CAM_EUL run with identical conditions of HS forcing, no topography, and climatological initial conditions. The output data were time averaged over the last 750 days of the integrations. The abscissa scale describes latitude; the ordinate scale shows pressure levels (hPa) on the left and height (km) on the right.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Distributions of (left) *temperature variance* and (right) *meridional wind variance* from both (top) CAM_SEM and (bottom) CAM_EUL run with identical conditions of HS forcing, no topography, and climatological initial conditions. The output data were time averaged over the last 750 days of the integrations. The abscissa scale describes latitude; the ordinate scale shows pressure levels (hPa) on the left and height (km) on the right.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Distributions of (left) *temperature variance* and (right) *meridional wind variance* from both (top) CAM_SEM and (bottom) CAM_EUL run with identical conditions of HS forcing, no topography, and climatological initial conditions. The output data were time averaged over the last 750 days of the integrations. The abscissa scale describes latitude; the ordinate scale shows pressure levels (hPa) on the left and height (km) on the right.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Distributions of (top) annual average zonal wind (m s^{−1}) and (bottom) temperature fields (K) from integration output of CAM_SEM, CAM_EUL, and corresponding NCEP reanalyses for the full physics experiment. CAM_SEM was run with identical conditions to those of an archived version of CAM_EUL (CAM2.0.1). Coordinate axes are identical to Fig. 4.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Distributions of (top) annual average zonal wind (m s^{−1}) and (bottom) temperature fields (K) from integration output of CAM_SEM, CAM_EUL, and corresponding NCEP reanalyses for the full physics experiment. CAM_SEM was run with identical conditions to those of an archived version of CAM_EUL (CAM2.0.1). Coordinate axes are identical to Fig. 4.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Distributions of (top) annual average zonal wind (m s^{−1}) and (bottom) temperature fields (K) from integration output of CAM_SEM, CAM_EUL, and corresponding NCEP reanalyses for the full physics experiment. CAM_SEM was run with identical conditions to those of an archived version of CAM_EUL (CAM2.0.1). Coordinate axes are identical to Fig. 4.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Distributions of average annual zonal wind and temperature fields for CAM_SEM run (bottom) with a uniform grid and (top) using LMR (denoted as LMR_USA) as seen in Fig. 3 (top). Forcing and initial conditions were the same as those used for Fig. 5, as are the coordinate axes.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Distributions of average annual zonal wind and temperature fields for CAM_SEM run (bottom) with a uniform grid and (top) using LMR (denoted as LMR_USA) as seen in Fig. 3 (top). Forcing and initial conditions were the same as those used for Fig. 5, as are the coordinate axes.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Distributions of average annual zonal wind and temperature fields for CAM_SEM run (bottom) with a uniform grid and (top) using LMR (denoted as LMR_USA) as seen in Fig. 3 (top). Forcing and initial conditions were the same as those used for Fig. 5, as are the coordinate axes.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Sea level pressure distributions in earth’s latitude (ordinate) and longitude (abscissa) for the models CAM_SEM (uniform grid) and LMR_USA described for Fig. 6.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Sea level pressure distributions in earth’s latitude (ordinate) and longitude (abscissa) for the models CAM_SEM (uniform grid) and LMR_USA described for Fig. 6.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Sea level pressure distributions in earth’s latitude (ordinate) and longitude (abscissa) for the models CAM_SEM (uniform grid) and LMR_USA described for Fig. 6.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Same as Fig. 7, but for the total precipitation rate (mm day^{−1}). The domain presented is local to the continental United States (used for LMR).

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Same as Fig. 7, but for the total precipitation rate (mm day^{−1}). The domain presented is local to the continental United States (used for LMR).

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Same as Fig. 7, but for the total precipitation rate (mm day^{−1}). The domain presented is local to the continental United States (used for LMR).

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Results from the AMIP integration experiment showing the annual radiative surface temperature (ordinate in K) as a function of latitude (abscissa), comparing (a) CAM_SEM with CAM2 and (b) CAM3 with NCEP reanalysis data.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Results from the AMIP integration experiment showing the annual radiative surface temperature (ordinate in K) as a function of latitude (abscissa), comparing (a) CAM_SEM with CAM2 and (b) CAM3 with NCEP reanalysis data.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Results from the AMIP integration experiment showing the annual radiative surface temperature (ordinate in K) as a function of latitude (abscissa), comparing (a) CAM_SEM with CAM2 and (b) CAM3 with NCEP reanalysis data.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Same as Fig. 9 but for the annual total cloud cover (in percent).

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Same as Fig. 9 but for the annual total cloud cover (in percent).

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

Same as Fig. 9 but for the annual total cloud cover (in percent).

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

The annual vertical distributions of temperature and zonal wind with latitude for (left) CAM_SEM and (right) CAM2 for the AMIP integration experiment.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

The annual vertical distributions of temperature and zonal wind with latitude for (left) CAM_SEM and (right) CAM2 for the AMIP integration experiment.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

The annual vertical distributions of temperature and zonal wind with latitude for (left) CAM_SEM and (right) CAM2 for the AMIP integration experiment.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

The annual distribution of sea level pressure (hPa) on lat–lon maps from CAM2 and CAM_SEM for the AMIP integration experiment.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

The annual distribution of sea level pressure (hPa) on lat–lon maps from CAM2 and CAM_SEM for the AMIP integration experiment.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

The annual distribution of sea level pressure (hPa) on lat–lon maps from CAM2 and CAM_SEM for the AMIP integration experiment.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

The annual distributions of total cloud cover (in percent) on surface maps for CAM_SEM, CAM2, CAM3, and observational data (Warren et al. 1986, 1988) for the AMIP integration experiment.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

The annual distributions of total cloud cover (in percent) on surface maps for CAM_SEM, CAM2, CAM3, and observational data (Warren et al. 1986, 1988) for the AMIP integration experiment.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1

The annual distributions of total cloud cover (in percent) on surface maps for CAM_SEM, CAM2, CAM3, and observational data (Warren et al. 1986, 1988) for the AMIP integration experiment.

Citation: Monthly Weather Review 134, 12; 10.1175/MWR3360.1