1. Introduction: Model accuracy and computational efficiency for exascale computing
Numerical algorithms used for atmospheric general circulation modeling have undergone constant evolution over the past half century (e.g., World Meteorological Organization 1969; Williamson 2007; Lauritzen et al. 2011). Early models, such as the Mintz–Arakawa model (see Gates et al. 1971), used the finite-difference method (FDM) on a latitude–longitude grid; both the Global Forecast System (GFS) atmospheric model (Sela 1980) and the European Centre for Medium-Range Weather Forecast’s high-resolution model (Simmons et al. 1989) used spectral methods (e.g., Machenhauer 1979). Recent models, such as the Finite-Volume Cubed-Sphere Dynamical Core (FV3; Lin 2004), the Model for Prediction Across Scales (MPAS; Skamarock et al. 2012), the Nonhydrostatic Icosahedral Atmospheric Model (NICAM; Satoh et al. 2008), the Nonhydrostatic Icosahedral Model (NIM; Lee and MacDonald 2009), the Flow-Following Finite-Volume Icosahedral Model (FIM; Bleck et al. 2015), and the Icosahedral Nonhydrostatic model (ICON; Wan et al. 2013; Zängl et al. 2015), have adopted gridpoint models based on the finite-volume (FV) scheme. In addition, the spectral-element method (e.g., Maday and Patera 1989) has been implemented in atmospheric models (e.g., Taylor et al. 1997; Giraldo et al. 2013; Choi and Hong 2016). Considering ambitious goals, such as climate simulation at 1-km scale (Voosen 2018) and real-time global weather predictions at subkilometer resolution with extension capability for subseasonal and seasonal predictions, today more than ever, stringent requirements are imposed on model accuracy, stability, scalability and suitability to run efficiently on millions of parallel computing units, a likely characteristic of future exascale machines.
Despite having at best a first-order accuracy with respect to the control volume [i.e., ~O(N−1), where N is the number of quasi-uniform grid points (or grid cells) on the sphere], the finite-volume method (FVM) has been a mainstream choice for modeling on quasi-uniform grids at high resolution owing to its simplicity and computational efficiency. In contrast, spectral models are more accurate but limited by computational efficiency especially on large-scale parallel machines. The hindrance lies in that Legendre transforms and Fourier transforms therein needs global information from each latitude, but its collection is inherently slow on distributed memory systems [see recent progress in O’Neil et al. (2010) and Wedi et al. (2013)]. Gridpoint methods, such as the icosahedral-grid-based FVM approach, provides a compromise between accuracy and computational cost—having low order of accuracy yet free from the costly global transformations between real and reciprocal space. Advantages of icosahedral models include the lack of pole-problems, quasi-homogeneity in cell volume, the benign transition from the 12 singular pentagons to their hexagonal neighbors and beyond, and straightforward algorithms for grid generation, optimization, stretching, and variable resolutions (e.g., Du et al. 1999; Tomita et al. 2001; Wang and Lee 2011; Miura and Kimoto 2005; Rauscher and Ringler 2014).
Having access to higher resolution via FVM could potentially put an end to parameterizations of subgrid dynamic processes that were required before (e.g., Satoh et al. 2008). However, this advantage needs to come with meticulous design of numerical algorithms for accuracy and stability purposes, especially in curvilinear space. Take surface topography as an example. Given the same complex terrain resolved at meter resolution, according to fractal theory (Mandelbrot 1982), the nonlinearity exhibited at subkilometer scales will be much stronger than at 100-km scales. Hence, the existence of pronounced nonlinearity in the dynamic equation at high resolution poses a strong challenge to the stability and accuracy of FVM, even if satisfactory performance had been achieved for idealized test cases (e.g., Williamson et al. 1992).
In our view, because of the complexity of FVM for irregular grids on sphere, model stability can hardly be separately discussed without studying accuracy issues. For example, vector field interpolation on sphere as well as vector field reconstruction from its normal component are ill-defined mathematical problems, yet they are needed by FVM (e.g., Peixoto and Barros 2014). How robustly a model can withstand a long time integration without resorting to explicit numerical damping yet still producing reliable results appears to be an important test stone in our opinion, even though it is admitted that numerical diffusion (or damping) is a part of the model numerical scheme. In this paper, we will discuss separately simulations due to the pure FV algorithm design and the effect of further imposing explicit diffusion.
Shallow-water model solvers based on icosahedral geodesic grids come in many flavors. Some of the early works include Vestine et al. (1963), Sadourny et al. (1968), Williamson (1968), and Cullen (1974). These solvers can differ in variable-staggering schemes, local map projections (e.g., Phillips 1973), and variable interpolation methods. Among the variable-staggering schemes introduced by Winninghoff (1968) and Arakawa and Lamb (1977), the A-grid scheme refers to the method in which prognostic variables are collocated at the center of a grid cell, while in the C-grid scheme, only mass field variables reside at the cell center while the normal components of velocity are specified on cell edges. Examples of software implementations for FV icosahedral schemes include models on A-grid (NICAM, NIM, and FIM), C-grid (MPAS), Z-grid, ZM-grid, and SB-grid (Randall 1994; Heikes and Randall 1995a; Ringler and Randall 2002; Xie 2019; Miura 2019), and others (Stuhne and Peltier 1996, 1999).
To design and develop more advanced FV schemes, it is important to understand in detail the current available methods, such as the A-grid scheme from Tomita et al. (2001) and C-grid schemes from Thuburn et al. (2009) and Ringler et al. (2010) (also known as the TRSK scheme). To allow for a pure algorithm level comparison related to FV, we undertake a software implementation task, programming independently the grid generators, FV algorithms, and test cases. This approach avoids biases introduced in different numerical damping, time integration methods, and qualities of icosahedral grids. We also avoid methods to advect potential vorticity as an independent dynamic constraint, since our goal is to compare the effect due to the basic grid level algorithm design alone. We test model accuracy and numerical stability for very long integration times far beyond standard test requirements, which reveals some important numerical issues in both A-grid and C-grid. In this sense, our study exemplifies the current status of the FVM for nonlinearity and long time integration, which is a necessary step to search for improved algorithms. We wish to stimulate research interest in numerical algorithm development for the nonlinear dynamics on the sphere using FVM. This work is part of the Exascale Project at NOAA/ESRL/GSL, which seeks to evaluate and quantify scientific accuracy and computational efficiencies of numerical approaches. One of our ongoing efforts is to evaluate and compare the computational efficiency of different shallow-water model schemes, especially the communication performance at cloud-permitting scales (3 km) in global models.
The paper is organized as follows: section 2 gives an overview about technical details on variable staggering and grid generation, which is followed by simulation results on five Williamson test cases and a numerical operator accuracy test in section 3. In section 4, we show a new method to reduce the systematic error on C-grid, which combines the Coriolis term with kinetic energy gradient for a linear combination operation. Two numerical tests are performed to reveal pros and cons of our method, which are shown in conjunction with the method of Peixoto (2016). The effects of diffusion on the A-grid and C-grid simulations are compared in detail in section 5. Section 6 shows our preliminary work on software optimization and parallelization of the A-grid and C-grid codes. Conclusions from this study are drawn at the end.
2. Overview of the A-grid and C-grid staggering methods and some technical details in the code
The FVM in conjunction with the icosahedral grid is used to solve the shallow-water model on the sphere, which comprises the continuity and momentum equations:
Here τ denotes time and we have reserved subscript t to indicate the tangential component of a vector (e.g., ut is the tangential velocity component, which plays an important role in formulating the C-grid scheme). The symbol h* represents the fluid depth, V is the velocity vector, g is the gravity, and hs isthe underlying topography; f = 2Ω sinθ is the so-called Coriolis parameter, and ξ = k ⋅ ∇ × V is relative vorticity. The FVM is used to approximate spatial derivatives of the fields. We use the fourth-order Runge–Kutta method (Press et al. 1992) to evolve the flow in time. The time steps we choose for the simulation lie between those of NICAM and MPAS [i.e., 480, 240, and 120 s for grid level 4, 5, and 6, respectively]. This set of time steps is used for both A-grid and C-grid calculations.
Winninghoff (1968) and Arakawa and Lamb (1977) introduced several schemes to stagger field variables (h and V). Among them are the A-grid scheme with prognostic variables (h and V) positioned at cell centers, and the C-grid scheme in which the two prognostic variables are h and the normal component of the velocity un, with the former located at cell centers and the latter centered on cell edges. In the A-grid scheme, prognostic variables at cell centers are interpolated to edges and then used in line integrals to compute gradient, divergence, vorticity, and so on. An example is shown in Fig. 1a for vector interpolation on a square grid, where a vector positioned at the middle of an edge is obtained by averaging values from two adjacent square centers. In the C-grid scheme, since the Coriolis force term, −(f + ξ)ut, is needed in the dynamic equation for un, either the tangent ut or the whole Coriolis force term has to be interpolated from prognostic variables un and h. The interpolation method would couple the icosahedral grids, the variable staggering pattern, and the dynamic equations. On a square lattice with C-grid staggering (Fig. 1b), the tangential vector component ut on an edge is derived by averaging the normal vector component un from four adjacent edges [ut = 0.25(u1n + u2n + u3n + u4n)]. The interpolation method on icosahedral grids can be viewed as an extension from square grids. For the A-grid, velocity on an edge is determined from a weighted average of four hexagon centers (see formula in Fig. 1c). In the C-grid scheme, ut on an edge is written as a linear combination of un values from 9 or 10 adjacent hexagon edges (see formula in Fig. 1d). The FV formula for calculating a divergence field through line integrals is also shown in Fig. 1. More details on the linear combination coefficients for velocity interpolation and the FV discretization can be found in previous publications on the method for NICAM and MPAS (Tomita et al. 2001; Thuburn et al. 2009; Ringler et al. 2010).
Illustration of the finite-volume algorithm applied to (a) the latitude–longitude grid using the A-grid staggering scheme, (b) the latitude–longitude grid using the C-grid staggering scheme, (c) the icosahedral grid using the A-grid staggering scheme, and (d) the icosahedral grid using the C-grid staggering scheme. Note that the
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
Some background information on icosahedral grids is provided here to facilitate understanding the terminology of the grid level and resolution to be used in the following section. A computer code is written to generate icosahedral grids that incorporates the spring dynamics method for the A-grid (Tomita et al. 2001) and the centroidal Voronoi tessellations (CVT) method from Du et al. (1999) for the C-grid. Grid quality is strictly controlled. Force on each grid point is converged to 10−6–10−7 using spring dynamics [Eq. (6) in Tomita et al. 2002]. Ratio of the maximum gridpoint shift distance from two consecutive bisection procedure relative to average gridpoint distance is set to 10−13 for the CVT grid. The velocity interpolation coefficient
3. Williamson test cases: A-grid (Tomita et al. 2001) versus C-grid (TRSK scheme)
Five representative test cases (TC) were selected from the standard test set devised by Williamson et al. (1992). Among them, three have analytic solutions: simple steady zonal flow (TC2), flow with compact support (TC3), and forced nonlinear system with a translating low (TC4). The other two nonlinear cases include the mountain-wave test (TC5) and the Rossby–Haurwitz wave test (TC6). TC1 is not performed because of its simplicity, which tests solvers only for advection [Eq. (1)] but not for the Euler equation [Eq. (2)]; TC7 is not performed because of lack of public access to the filtered initial conditions without normal modes. The initial velocity fields for TC2 and TC3 contain low-order spherical harmonics, while the other cases (TC4–6) involve terms with high-order spherical harmonics, which are prone to generating high wavenumber signals. The numerical advantage and deficiency of the A-grid and C-grid staggering schemes were exposed by long time integration without numerical damping. All other parameters were kept identical for comparison during simulations, such as time step and time integration method. The effect of numerical diffusion on the A-grid and C-grid results will be discussed in section 5.
a. Large-scale zonal geostrophic flow (TC2)
In TC2, the steady-state zonal flow is balanced by the geostrophic height field satisfying the shallow-water equations:
with ||r||2 = a (Earth radius), ω and Ω being the angular frequency vector of the zonal flow and the sphere, respectively, and h0 being a constant (e.g., 3 km). Since zonal velocity u ~ cosθ
So far only the C-grid energy conserving scheme by Ringler et al. (2010) has been shown to withstand a 1-yr simulation in TC2 without explicit damping; the A-grid scheme from Tomita et al. (2001) was shown to withstand a 90-day integration without damping. A 1-yr simulation at various grid resolutions was run to verify the previous results. Striking in Fig. 2a is the C-grid error norm in the depth field [L∞(h) and L2(h)], remaining steady over a year with constrained fluctuations when tested at grid level 5 (G5; ~240-km resolution). The error norms from the A-grid method begin to diverge at ~200 days, and their fluctuation amplitude is larger than the C-grid results. In the original work by Tomita et al. (2001), the model was run for 90 days without damping; here we are able to maintain numerical stability for 200 days without explicit damping, which is likely due to minor coding and grid optimization differences. On the A-grid, the spring dynamics method used for grid optimization plays an important role in reducing numerical noise, and the simulation results would be worse had other grid generation methods been used, such as the standard grid or the recursive gravitational-center grid [not shown here; see Tomita et al. (2001) for reference].
Test case 2 (TC2: steady-state geostrophic flow): numerical accuracy and time integration stability for the A-grid and C-grid schemes without explicit numerical damping. (a) Time evolution of the L2 and L∞ norms for depth up to 365-day integration for the two staggering schemes. (b) The convergence rate of the L2 and L∞ norms with respect to the number of icosahedral grid points (or the gridpoint level, G4–G7) at the end of the 50-day integration. Reference lines shown in black are for the first-order accuracy [O(N−1)] and second-order accuracy [O(N−2)], with N being the number of icosahedral cells.
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
Figure 2b shows the convergence rate of these error norms with increasing grid resolution from grid level 4 (G4; 480-km resolution) to grid level 7 (G7; 60-km resolution) at the end of a 50-day simulation. In the A-grid method, both L∞(h) and L2(h) norms decrease linearly with the number of grid points. On the C-grid, this is true only for the L2(h) norm. The L∞(h) norm barely decreases with grid resolution. This is a known imperfection in the TRSK scheme. Some discussion will be given in section 3f. Also shown in Fig. 2 are two reference lines illustrating first-order [O(N−1)] and second-order accuracy [O(N−2)] (N being the number of grid points or hexagonal cells). Note that since N1/2 ~2πa/(Δx) on icosahedral grids (a being Earth’s radius), the O(N−1) and O(N−2) are equivalent to O(Δx2) and O(Δx4), respectively, in terms of the one-dimensional latitude–longitude grid length. Here we adopt the O(N−1) notation. Both A-grid and C-grid methods show at best first-order accuracy, because of the FVM used on an irregular icosahedral grid, which is consistent with the previous study (Tomita et al. 2001). Yet, the A-grid shows faster L2 norm convergence rates than C-grid beyond grid level G5. The error convergence rates with increasing number of grid points for other test cases are expected to be worse than for TC2, since TC2 is among the least nonlinear test cases. Therefore, we will not show the O(N−1) and O(N−2) reference lines in figures for other test cases.
To understand the nature of the nondecaying L∞(h) norm, we plot in Fig. 3 the probability density function (PDF) of the pointwise relative error norm, which is defined as
Probability density function of the relative error in fluid depth (|h − hT|/hmax) in test case 2 due to the A-grid and C-grid schemes at two different grid levels: (a) G5 (250-km resolution) and (b) G7 (60-km resolution).
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
b. Steady-state nonlinear zonal geostrophic flow with compact support (TC3)
The initial condition for the flow with compact support in TC3 is shown in Fig. 4a, where the sharply confined zonal wind velocity (red line) balances the sharp reduction in geopotential at northern midlatitudes. “Compact support” refers to the stagnant (zero velocity) portion of the fluid in the Southern Hemisphere (h = 3000 m). The steady-state solution in TC3 permits an easy measure of the error norms for the two gridpoint methods. Like in TC2, the A-grid method is effective in controlling error norms in TC3 at resolutions from G4 to G7—the L2 and L∞ norms for the height field decrease linearly with increasing number of icosahedral grid points (red lines in the log–log graphics in Fig. 5a). In the C-grid method, only the L2(h) error norm is consistent with the first-order accuracy characteristics of the FVM; the L∞(h) norm shows clear reduction from grid level G4 to G5, but not at higher resolutions (above G6, see the blue squares in Fig. 5a). In a previous study by Stuhne and Peltier (1999), an increase in the L2(h) and L∞(h) errors was observed from grid level G6 to G7 due to numerical inaccuracy. In the work by Heikes and Randall (1995a), the error norms in L2(h) and L∞(h) were only reported at G5 and G6 resolution. Their convergence rate with respect to the number of grid points is less than first order. TC3 results were not shown in Ringler et al. (2010).
Initial conditions for (a) test case 3 and (b) test case 4: the geopotential height h and the zonal velocity u (when the longitude λ = 0 in TC4) as a function of latitude θ.
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
The L2 and L∞ error norms for the h field (fluid depth) as a function of gridpoint resolution for (a) test case 3 and (b) test case 4 at day 6.
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
c. Forced nonlinear system with a translating low (TC4)
TC4 is less frequently demonstrated in gridpoint models than in spectral models [e.g., it was omitted in Tomita et al. (2001) and Ringler et al. (2010)]. Heikes and Randall (1995a) only briefly covered this case, without showing its truncation error convergence with increasing grid resolution. In TC4, two westerly jets with zonal velocity u = 20 sin14(2θ) m s−1 centered at latitude θ = ±π/4 are superimposed upon a translating low pressure center, which has a period of 23.17 days. Figure 4b shows the initial condition for depth and zonal velocity across the λ = 0 plane (0° longitude) as a function of latitude. The vortex circulation is manifest from the dip in the fluid depth field (the minimum at θ = π/4) and a sign change in the superimposed velocity direction near θ = π/4 (Fig. 4b). Simulations were run for 6 days with u0 = 20 m s−1, and the contrast in the error analysis between the two methods is shown in Fig. 5b. In the C-grid method, both L2(h) and L∞(h) error norms fail to decrease with grid resolution, which is a problem in the C-grid scheme and will be discussed in section 3f. Such problems do not arise in the A-grid method.
Note that recent simulations using a modified TC2 with a steady thin-layer also revealed the nonconvergence problem of the TRSK scheme at high resolution for both L∞ and L2 norms (see Fig. 9 in Peixoto 2016), which is consistent with our results here on TC4 (Fig. 5b).
d. Zonal flow over an isolated mountain (TC5): Strongly nonlinear test case
The spatial gradient term acting on the mountain height field in TC5 is a major source generating nonvanishing high wavenumber components in Fourier space. Simulations from the A-grid and C-grid methods are compared with the baseline solution from the spectral transform model with T511 spectral truncation (Jakob et al. 1993). The standard test with a 15-day integration (Fig. 6, left column) indicates the A-grid and C-grid results are comparable, except that the position of the 5900-m isoline (near equator) in the A-grid model appears closer to the baseline solution than does the C-grid. After 90 days, the height field from the initial zonal flow breaks into multiple troughs (Fig. 6, right column). There remains some correspondence in the trough locations between these three methods. The A-grid solution becomes noisy on each grid point, although resemblance to the baseline still exists. The C-grid result is free from notable gridpoint noise, but some large errors appear near the North Pole. This is likely due to the phase mismatch between the velocity and the height field equations during the time integration, since we did not implement the anticipated potential vorticity method (Sadourny and Basdevant 1985), which was shown to be useful in improving the C-grid results for TC5 (Ringler et al. 2010), although no theoretical proof has been provided.
Test case 5: flow over an isolated mountain. Shown is the height field at (left) 15 and (right) 90 days calculated at grid level 6 (120-km resolution) using algorithms based on (top) the A-grid, (middle) the C-grid, and (bottom) the spectral transform model, respectively, without explicit numerical damping.
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
The L2(h) and L∞(h) norms generated from the A- and C-grid were shown in Fig. 7, where we find sublinear convergence [(~O(N−1)] in error norm with respect to the number of grid points. Our C-grid results shown in Fig. 7 are equivalent to results from Fig. 8 in Ringler et al. (2010) [e.g., at G6 resolution, the L2(h) and L∞(h) norms from our study are 2.0 × 10−4 and 1.5 × 10−3, respectively, which agree well with their results, 3.5 × 10−4 and 2.0 × 10−3]. Furthermore, we calculated the fractional error for total energy and potential enstrophy, defined as the ratio of the absolute deviation from its initial value (see Fig. 8). Remarkable total energy conservation results are obtained from the C-grid scheme, the error being about a factor of 10−3 smaller than the A-grid fractional error in energy (Fig. 8a). However, for potential enstrophy, the C-grid energy-conserving scheme gives slightly worse results than the A-grid method. Our results are consistent with Fig. 17 in Tomita et al. (2001) and Fig. 9 in Ringler et al. (2010). Moreover, as an illustration of a faithful implementation of the C-grid scheme in our software, we show in Fig. 9 the relative error in potential enstrophy as a function of time calculated for several grid resolutions (G4–G7), which is slightly better than the errors shown in Fig. 9 of Ringler et al. (2010) (i.e., error from this implementation at day 15 is one-half of their results at every resolution, which we ascribe to minor difference in numerical coding).
The L2 and L∞ error normals for the h field as a function of gridpoint resolution for test case 5 after 15-day integration.
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
Test case 5: Time evolution of the fractional error in (a) total energy and (b) potential enstrophy for the two schemes up to 15 days; TE and PE stand for total energy and potential enstrophy, respectively, at the current time, and TE0 and PE0 stand for total energy and potential enstrophy, respectively, at the initial time.
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
Test case 5: Time evolution of the fractional error in potential enstrophy from the C-grid calculation at various grid levels from G4 (480-km resolution) to G7 (60-km resolution).
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
Information from Figs. 2, 5, and 7 show that for the C-grid the L∞(h) norm convergence with respect to the number of icosahedral grid points exhibits sublinear behavior [worse than O(N−1)] in TC5, but fails to decrease in TC2–4. We realize that aliasing errors in the discrete Fourier transform when interpolating a scalar field (e.g., h) from the original spectral grid (uniform longitude and Gaussian latitude) to the irregular icosahedral grid should not be neglected in the case of T511 resolution. We suspect it is the error cancellation between the aliasing error and the intrinsic error in the C-grid scheme that yields the sublinearly decreasing behavior of the L∞ norm with grid resolution in TC5. The true C-grid error for TC5 is still unknown. Therefore, we recommend using TC2–4, which have analytical reference solutions, to study numerical error convergence with respect to grid resolution.
e. Rossby–Haurwitz waves test (TC6): Weakly nonlinear test case
An analytical expression for a linearized two-dimensional vortex equation on sphere for a nondivergence flow was derived by Haurwitz (1940), which, of course, is not a solution for the shallow-water model on sphere. It was used as an initial condition in the previous simulation by Phillips (1959). More thorough studies showed that the wavenumber R = 4 pattern can be relatively stable until a breakdown happens between 25 and 80 days depending on the specific model being used (e.g., spectral methods or FVM; Thuburn and Li 2000). In our work, R = 4 is the parameter that we used. A comparison between the A-grid and C-grid solutions is shown in Fig. 10 using results from the spectral transform model with T511 resolution as the baseline (Jakob et al. 1993). In the 14-day standard test (Fig. 10, left column), the A-grid results at G6 resolution contain an erroneous deviation (a mild trough) in the 8400-m isoline shape, particularly at 110°E and 70°W. The C-grid results are more consistent with the baseline. When simulations are extended to 60 days, only the spectral model retains the wavenumber R = 4 character with nodes and peaks in the height field located on the equator. Note that insufficient wavenumber cutoff, such as T42 in the spectral model (τ = 600 s), will produce spurious solutions at 60 days, as shown in Fig. 6.11(b) of Jakob et al. (1993). Without using numerical damping, both gridpoint models fail to preserve the Haurwitz wavenumber R = 4 structure at 60 days; instead, the peaks in the height field migrate off the equator. Fast nonlinear oscillations appear in the A-grid solution on each isoline due to gridpoint noise contamination. These spurious oscillations are absent in the C-grid staggering, as also seen in TC5.
Test case 6: Rossby–Haurwitz wave. The height fields calculated at (left) 14 and (right) 60 days using the (top) A-grid method and (middle) C-grid method (without explicit numerical damping) are compared with the baseline solution from the (bottom) spectral method (T511 with damping).
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
f. Accuracy in computing the Coriolis force term on the sphere
Next we conduct an experiment to test the accuracy of each leading component in the tendency function [right-hand side of Eq. (2)] for the A-grid and C-grid schemes. The chosen test functions are h = h0 cos(mλ) cos4(nθ) and V = u0 sinλ∇[cos(mλ) cos4(nθ)] (Heikes and Randall 1995b) with m = n = 3. The analytical expressions for their spatial derivatives are given in appendix B and used as a baseline to reveal the numerical errors from FVM. Figure 11 displays on the sphere the absolute errors from the two methods for a few quantities related to spatial derivatives: height gradient, divergence, vorticity, and the Coriolis term. The error for a vector is defined as the vector two-norm at each grid point (appendix A). For the C-grid, the normal velocity (un) staggered at the middle of the edge can be used conveniently to calculate divergence (flux) and vorticity (circulation). In addition, the CVT grid in the C-grid scheme provides convenience for calculating the height (pressure) gradient term using FDM. Here, we find similar yet still distinguishable accuracy between the A-grid and C-grid methods in computing ∇h, ∇ ⋅ (hV), and ∇ × V (Fig. 11) (e.g., at local scale the gradient and divergence terms from the C-grid method are slighter more accurate than the A-grid results). For the Coriolis term (f + ζ)k × V in the momentum equation, however, the error for the A-grid is about two orders of magnitude smaller than for the C-grid. This is due to the use of the linear combination weights [defined in Thuburn et al. (2009)] that ensures steady geostrophic modes and energy neutrality from the Coriolis force in linear equations. The drawback is to sacrifice the local precision on hexagon edges. This is likely a contribution to the nonconvergence of the L∞(h) at high grid resolution as shown in test cases 2, 3, and 4.
Absolute errors in a few spatial operators for the velocity and depth field defined in appendix B [(left) gradient ∇h, (left center) divergence ∇ ⋅ (hV), (right center) vorticity ∇ × V, and (right) Coriolis force term (f + ζ)k × V], calculated on the sphere from the (top) A-grid and (bottom) C-grid methods at icosahedral grid level G4 (480-km resolution).
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
4. Alternative methods and thoughts to improve the C-grid scheme
Recently Peixoto (2016) shows that the nonconvergence problem of the L∞(h) error norm with respect to the number of grid points in the TRSK scheme can be removed in TC2 if three modifications are applied (the MODF-HCM-SCVT method in Peixoto 2016): (i) velocity vector reconstruction using the method of Perot (2000), (ii) an alternative Coriolis force term formulation, and (iii) barycentric interpolation to improve the ∇ ⋅ (hV) term. The method by Peixoto (2016), however, degrades the numerical stability. On the other hand, the methods proposed by Gassmann (2013, 2018) shows promise to alleviate the Hollingsworth instability (Hollingsworth et al. 1983) present in the C-grid model. Here we first analyze the consistency between the dynamic equation and the velocity linear interpolation method, then propose a method that treats the Coriolis force term and kinetic energy gradient on an equal footing.
a. A CD-grid concept to derive a Coriolis-force-term formulation analogous to the TRSK method
One of the key concepts in the TRSK scheme is the Coriolis force term reconstruction. It is analogous to the velocity interpolation method, yet its proof is not explicit. An essential property that contributes to the energy conservation is the antisymmetry of the
where η = f + ξ,
with le and de being the length of the cell edge e and the distance between two cell centers sharing the edge, respectively. We notice that the factor
Consider the following. If dynamic equations for the two velocity components un and ut were to be evolved (or integrated) simultaneously, then the Coriolis force would not contribute to the work at each local point. The Euler equations are
Here τ denotes time; n and t denote, respectively, normal and tangential components;
The assumption underlying the C-grid algorithm is that the tangential and normal components of a vector field on a cell edge can respectively be expressed as a linear combination of the normal and tangential components along the perimeter of two adjacent cells, be it velocity, gradient force, or any other vector field. Instead of using Eq. (5), here we start with its inverse representation:
with e and e′ representing coupled edges from two adjacent hexagons, and
Next we take partial time derivative on both sides of Eq. (8), then substituting Eqs. (6) and (7) into the resulting equations, we get (assuming
Apparently, this equation can be satisfied if the Coriolis force term and the pressure gradient term independently satisfy the above interpolation formula on each edge of the hexagon, e:
Here Eq. (10) belongs to the definition of the normal and tangential component interconversion relationship for vector fields; Eq. (11), which is similar to Eq. (4) of the TRSK scheme could be used as the derived Coriolis force term to be inserted into Eq. (6) to evolve un in time. The proof that
Note that in deriving Eq. (9) we have combined the velocity reconstruction relation [Eq. (8)] with the un–ut dynamics [Eqs. (6) and (7)], which is reminiscent of a CD-grid approach. Thus, the local relation V ⋅ (ηk × V) = 0 would hold on each cell edge and the Coriolis force would be neutral to energy generation. In practice, only the un dynamics equation [Eq. (6)] is evolved in time with the help of the Coriolis force term [e.g., Eq. (4) in TRSK]. We also note two facts: (i) as long as Eq. (9) is satisfied, the proper time evolution of ut required by Eq. (7) is guaranteed; hence Eq. (7) does not need to be solved in the C-grid algorithm as well known (otherwise, the method is called CD-grid); (ii) the functional form for kinetic energy K will not affect the consistency between un and ut provided that the gradient term and the Coriolis force term can be decoupled in Eq. (9). Here we view Eq. (9) as a guide to understanding the self-consistency between un, ut, and their coupling relation.
b. Our method: Combine kinetic energy gradient with Coriolis force term for a linear combination operation
The Hollingsworth instability is a numerical instability mainly due to the numerical mismatch when summing the Coriolis force term and the kinetic energy gradient [i.e., (∇ × V) × V + ∇(1/2)V2] to recover the value of V ⋅ ∇V, especially for C-grid algorithms due to incomplete velocity (kinetic energy) information on cell edges (see Bell et al. 2017). The numerical noise therein has implications for baroclinic instability. We propose combining kinetic energy gradient with Coriolis force term before performing the linear interpolation:
This will treat the two terms in a numerical consistent manner. In the TRSK scheme, the Coriolis term is calculated from a linear combination process while the kinetic energy gradient is obtained using FDM. These two terms have different accuracy. The drawback of Eq. (12) is that we abandon the direct calculation of the normal component of kinetic energy gradient via FDM, but resorting to the linear combination of its tangent components via the
Comparison of numerical stencils used in the TRSK scheme vs this work, assuming the tendency (∂un/∂τ) is computed on an edge shared by two hexagons.
1) TC2: A large-scale geostrophic flow
Figure 12 compares the method proposed from this study [i.e., replacing Eq. (4) by Eq. (12) in the TRSK scheme] and the method of Peixoto (2016), in conjunction with the A-grid method of Tomita et al. (2001) and the C-grid TRSK scheme using TC2. Among the three C-grid approaches, the method of Peixoto (2016) generates the smallest error in both L2(h) and L∞(h) norm for the depth field (measured at day 12) and these error norms decrease linearly with the number of icosahedral grid points. Both the TRSK method and our method fail to converge the L∞(h) error norm at high resolution, even though our method does reduce the magnitude the TRSK L∞(h) norm without seriously degrading the stability of the original scheme. We find that in running TC2 the TRSK scheme is stable up to 2200 days, our method can withstand the test up to 300 days, the A-grid scheme can withhold a 200-day integration, while the method by Peixoto (2016) can withstand 20-day simulation even though its accuracy is the best among the four methods compared.
Test case 2: The error norms for the h field from four methods—the A-grid method of Tomita et al. (2001), the C-grid TRSK scheme, the method of Peixoto (2016), and the method proposed in this work [Eq. (12)]. Shown are (a) time evolution for the L∞(h) norm, (b) convergence of the L∞(h) norm with the number of icosahedral hexagonal cells, and (c) the convergence of the L2(h) norm. Simulations were run without damping on grid level 5.
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
2) TC2B: A thin-layer steady state to check baroclinic instability
Recently Peixoto et al. (2018) introduced a thin-layer test to reveal the stability of the C-grid scheme. This test is obtained, if the prognostic depth field in the Williamson TC2 is taken as a background terrain, upon which a thin layer (e.g., 0.01 m) is added as the true depth field while the velocity field remains the same:
Therefore, we call this thin-layer test TC2b. The variable definitions are the same as TC2 in Eq. (3).
Running this thin-layer test with fluid depth of 0.01, 1, and 100 m (Fig. 13) reveals that (i) the smaller the fluid depth the more severe this nonlinear instability will manifest itself on C-grid schemes, and (ii) the A-grid method is less susceptible to this instability test. Note that, when the fluid depth approaches zero (h* → 0), the full velocity tendency term 2Ω × V + V ⋅ ∇V + ∇(ghs) ≡ 0 in this test design. On C-grid, a major influence on the numerical error in this test lies in the V ⋅ ∇V term, which is the Hollingsworth instability, since the gradient term from the FDM contains much less error (see the ∇h term in Fig. 11). The A-grid stability for small fluid depths (h = 0.01 and 1 m; Figs. 13a,b) can be rationalized, because the kinetic energy is accurately defined at cell centers and the numerical treatment of the vorticity and the kinetic energy gradient terms has the same order of accuracy using the linear barycentric interpolation method. Its high accuracy is also enhanced by the homogeneity of the grid generated by spring dynamics.
TC2b: a steady-state thin-layer simulation with fluid depth (a) 0.01, (b) 1, and (c) 100 m, comparing the L2(h) error norm (RMS error) due to the A-grid method by Tomita et al. (2001), the C-grid TRSK scheme, and the method proposed in this study at grid level 5.
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
In Fig. 13c, at the finite fluid depth (h* = 100 m), the error growth on A-grid when compared with C-grid is severe when computed at grid level 5. Increasing the grid resolution does help with the A-grid error growth problem but is not a cure. For the case of h* = 100 m, we note that the error from the ∇(gh*) term begins to play a role. We attribute the A-grid error growth in this case to the general accuracy issue associated with the FVM on A-grid, since Fig. 11 shows that the numerical accuracy for the two operators [∇ ⋅ (hV) and ∇h] is slightly higher by the FDM on C-grid than the FVM on A-grid. We note that the emphasis of this test is on the small h* field (h* → 0), which singles out the numerical error contained in the V ⋅ ∇V term.
The method that we proposed fails to alleviate this instability on the C-grid. We speculate that having more accurate kinetic energy term [e.g., the work of Gassmann (2013), Skamarock et al. (2012), and Gassmann (2018)] and more accurate linear interpolation coefficients (the
5. The role of horizontal diffusion on model simulations
Numerical diffusion or damping, for example, using the second-order or fourth-order viscosity term, is an indispensable part of the model for long runs. We compare effects of horizontal diffusion on A-grid and C-grid simulations in detail, including which types of noise can be eliminated, how much diffusion is needed (i.e., what the effects of underfiltering and overfiltering are), how the outcome of second-order diffusion compares with fourth order, how accurate the numerical Laplace operators are on A-grid versus C-grid, and how diffusion affects energy conservation. Our major observations are summarized below.
The A-grid model stability can be extended in TC2 from about 200 days to over 3500 days when filtered with the fourth-order hyperviscosity term (Fig. 14). The temporal fluctuation pattern and amplitude for the L2(h) error norm (red lines) remain the same after filtering. As pointed out by Tomita et al. (2002), the Rossby wave error can be largely suppressed using the most homogeneous grid optimized by spring-dynamics, while gravity wave errors are insensitive to such optimizations. (We used β = 0.4 in the grid generator.)
Test case 2: Temporal variation of (a) the L2(h) error norm and (b) the L∞(h) norm with and without horizontal diffusion (damping) up to 5000 days at grid level 5. The hyperviscosity coefficients (fourth order) are 3.49 × 1014, 4.37 × 1013, 2.72 × 1012 m4 s−1 for e-folding time τ at 32, 256, and 4096 h, respectively.
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
Horizontal diffusion also extended the C-grid stability from about 2000 to 5000 days, although the TRSK scheme is already superb for stability without diffusion. Like the A-grid case, the computational modes on C-grid, probably due to the fivefold symmetry in grid structure, manifest itself in its original fluctuation amplitude. The slight growth in the mean value of L2 norm is attributed to either the intrinsic error in the FV algorithm on the C-grid or the numerical inaccuracy for the square of the Laplace operator on the CVT grid. Overdamping on the C-grid using e-folding time of 256 h can introduce larger errors than using 4096 h. We compared the discretization procedure used in calculating ∇4V by Tomita et al. (2001) on a spring dynamics grid with the one used by Skamarock et al. (2012) on a CVT grid. The latter uses ∇4V = ∇(∇2δ) − n∂t∇2ξ, with V, δ, and ξ being a vector, its divergence, and its vorticity, respectively, with t = r × n, and ∇2ξ being evaluated on hexagonal vertices. When compared with ∇4(ω × r) = 0, accuracy of the C-grid numerical operator for ∇4V is four orders of magnitude lower than the A-grid (10−22 vs 10−26 m−3 s−1), which is due to the nonoverlap of the midedges for hexagonal grid with its triangular dual grid (Table 2). The accuracy for the ∇2 operator is comparable among the two methods.
Comparison of the numerical accuracy of the Laplace operator on the A-grid and C-grid against analytic solutions, where r is the radial unit vector and ω = 6.06 × 10−5 s−1, as used in TC2 and TC5.
The gridpoint noise that plagued the A-grid long time integration results for TC5 has been effectively filtered out as shown in Fig. 15. To ensure smooth isolines for prognostic variables plotted on the sphere, the e-folding time for wavelength of 2 times the average gridpoint distance has to be less than 64 h at grid level 6 on the A grid. Nevertheless, the morphology of the isoline shapes is sensitive to the choice of e-folding time as shown in Fig. 15.
Test case 5 (A-grid damping): effect of varying e-folding time (EF) on the simulated depth field at 90 days using the fourth-order diffusion term at grid level 6.
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
Since simulations of TC5 using the C-grid algorithm barely show any gridpoint noise at 90 days (Fig. 6), the necessity to apply horizontal diffusion becomes nonintuitive. Here we compare results from second-order diffusion with fourth-order diffusion using an e-folding time of 4096 h, as it is unnecessary to apply heavy damping on C-grid like the one required by the A-grid. Results from Fig. 16 confirms that higher-order damping yields results with higher resolution since the filtering is more effective for large wavenumbers. The sixth-order diffusion is also commonly used in simulations (e.g., Tomita et al. 2002; Fig. 17).
Test case 5 (C-grid damping): Effect of (a) no diffusion vs the (b) second-order and (c) fourth-order diffusion on the depth field at 90 days with e-folding time of 4096 h at grid level 6.
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
Test case 5 (damping effect): flow over an isolated mountain. Shown is a comparison of the height field at 15 and 90 days calculated using the (top) A-grid, (middle) C-grid, and (bottom) spectral transform model (T213), respectively, with and without fourth-order diffusion. The e-foldings being used are 32 and 4096 h for the A-grid and C-grid, respectively, at grid level 6. The viscosities being used are 2.18 × 1013 and 1.71 × 1011 m4 s−1 for the A-grid and C-grid, respectively, at grid level 6. The diffusion coefficient used in T213 is 8 × 1012 m4 s−1.
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
When applying hyperviscosity to TC5 on the A-grid, we find our results comparable to the published one by Tomita et al. (2001). Even the C-grid and spectral model (T213) results show that horizontal diffusion has pervasive effects on small features. At 90 days, the morphology of the two valleys at 90°W and 50°N is also sensitive to filtering strength. The e-folding times used for the A-grid and C-grid at grid level 6 are 32 and 4096 h, respectively. In the T213 spectral model run, we used the hyperviscosity ν = 8 × 1012 m4 s−1 as in Jakob et al. (1993). Figure 18 compares damping effects (fourth order) on energy conservation. Longer e-folding time τef means less damping and less energy loss, but on the A-grid the weak damping (τef = 64 and 128 h) fails to produce less energy loss than using τef 32 h, which may be due to the residual numerical noise that is not filtered out that still damps out the energy. On C-grid, the total energy dissipation due to different e-folding times is well distinguished (Fig. 18), confirming the low intrinsic numerical noise owing to the TRSK scheme. The magnitude of the dissipation on the C-grid appears underestimated when compared with the A-grid (e.g., τref = 32 h), which is due to the error in the ∇4 operator (Table 2). Overall, these comparisons show that the A-grid scheme needs stronger filtering (larger diffusion coefficient) than the C-grid to eliminate noise and stabilize the solution for TC5, which means more energy loss and potentially more accuracy loss. This may have implications for climate simulations.
Total energy dissipation in test case 5 from the (a) A-grid vs (b) C-grid schemes affected by EF with fourth-order diffusion at grid level 6.
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
6. Progress in software optimization and parallelization of A-grid and C-grid codes
The single software framework provides a unique opportunity to compare and mutually enhance the optimization and parallelization of the A-grid and C-grid codes. The number of equations to be solved on both grids is roughly the same (4 times the number of icosahedral cells), yet the A-grid requires more operations for interpolation and calculating spatial derivatives, which results in 24% more floating-point operations on the A-grid than the C-grid (see Table 3). As CPU run time in serial and in parallel modes can be significantly affected by the software implementation process, a definite conclusion on timing comparison between the two algorithms is out of reach without qualifications. However, for our particular implementation, we find that the code speedup that results from changing the Intel compiler option from “-O0” (no optimization) to “-O3” (highly optimized) is 290-fold for A-grid and 77-fold for C-grid, which we attribute to better reuse of expressions for our A-grid code. Therefore, even though the A-grid has more floating-point operations at -O0, after -O3 optimization this A-grid code runs 67% faster than the C-grid code. Nevertheless, we are still exploring new ways to improve code optimization for both algorithms. Note that, for code parallelization and timing purposes, we add an artificial vertical dimension of 192 to the shallow-water model solver to mimic the three-dimensional effect.
Measure of the floating-point operations and CPU time for the A-grid and C-grid serial codes running TC2 using different compiler options within the same software framework.
The A-grid and C-grid modules in our framework have been parallelized using the Scalable Modeling System (SMS), a directive-based system utilizing MPI (Govett et al. 2003). We are able to obtain very good parallel efficiency (close to 1) on 16 000 CPU processors and on 800 GPU processors. More details will be given in a future publication that describes our software implementation.
7. Conclusions
This study implements and compares the shallow-water model solver from the NICAM model (A-grid) with the MPAS model (C-grid) in terms of accuracy, stability, error convergence with increasing resolution, conservation properties, and diffusion effects as summarized in Table 4. We emphasize the following:
Continued effort is needed to improve the current C-grid TRSK approach: the L2 error norm does not decrease with increasing resolution in TC4, besides the well-known L∞ norm problem and the issue of the Coriolis force term, even though long time stability and conservation property on the C-grid are exceptional.
The A-grid scheme passes the test cases without reservation, showing an excellent error decay rate with resolution. If no explicit damping is applied, however, the gridpoint noise on the A-grid contaminates the solutions in long runs. Stronger diffusion is needed on the A-grid than on the C-grid, which has drawbacks on model accuracy and conservation properties. After diffusion treatment, computational modes and relatively large error-fluctuation amplitude still exist on the A-grid, which are less prominent on the C-grid.
In the designed thin-layer geostrophic balance test (TC2b), the A-grid scheme shows smaller error and longer stability than C-grid methods, including the TRSK scheme, the method of Peixoto (2016), and the newly proposed method that combines the Coriolis term with the kinetic energy term. This is probably related to the Hollingsworth instability. These alternative methods on the C-grid are not recommended, because of the stability issue in the method of Peixoto (2016) and the L∞ error problem in the newly proposed method. Better improvement is still needed for the C-grid.
Summary of key observations from this comparative study; P denotes pass, F denotes fail, and Pr denotes pass with reservation. The reference figure numbers are given together with the rationale for the assignment [e.g., the nonconvergence of L2(h) and L∞(h) error norms with resolution].
Acknowledgments
We thank Drs. Yuanfu Xie and Duane Rosenberg for helpful discussion at the initial stage of this project and Drs. William Skamarock and Todd Ringler for helpful discussion on the MPAS shallow-water model scheme. We thank Dr. Almut Gassmann, Dr. Hilary Weller, and anonymous reviewers for critical reviews that significantly strengthened the paper. We also thank Dr. Rainer Bleck for proofreading the paper and polishing the language, Dr. Wei Yu for technical help, and Drs. Gerard Ketefian and Kayee Wong for helpful discussions. Author Yu is supported by funding from NOAA Award NA17OAR4320101. Authors Wang and Middlecoff are supported by funding from NOAA Award NA14OAR4320125. Author Peixoto is supported by FAPESP-Brazil funding under Grant 2016/18445-7.
APPENDIX A
Formulas for Global Error Norms
The formulas used in this work for the three global error norms—L1, L2, and L∞ for the scalar field h and the vector field v defined in Williamson et al. (1992)—are
Here the integral and max operators are defined on the sphere, dΩ = sinθdθdλ is the area element, and the integrals extend over the whole sphere (surface area 4π). The letter T denotes theoretical values. The dot product v ⋅ v is performed on a tangential plane of the sphere.
APPENDIX B
Formulas for Spatial Operators of the Test Function
Here we provide the analytic expressions for the spatial derivatives of the chosen height and velocity fields (h and V). They are used as baseline results to extract the numerical errors arising in the A-grid and C-grid schemes shown in Fig. 11:
where a is the radius of Earth, m and n are parameters, and the ∇ operator acts on fields located on the two-dimensional spherical surface. We used m = n = 3 in Fig. 11.
APPENDIX C
Proof for
Here we show that the inverse matrix for the coupling coefficients
where V1 = ∇χ and V2 = ∇ψ. Each of these three vectors (V, V1, and V2) can have its own orthogonal decomposition on the cell edge (i.e.,
Subtract the above two equations to obtain
Since u2n and u2t can also be connected by the inverse matrix elements
Hence
APPENDIX D
Energy Conservation from Using Eq. (12)
In Eq. (12), the new method that we introduced, the Coriolis force term was not symmetrized as in Eq. (4) (the TRSK method). Figure D1 shows the TC5 run using our method at grid level 6. The h field from the 15-day simulation is close to the TRSK results (top panel in Fig. D1). The total energy evolution with time shows significant deterioration from the C-grid TRSK method, whereas the potential enstrophy conservation is less affected since our method does not alter the method for computing vorticity and has slight impact on depth field (bottom panel in Fig. D1).
TC5: (top) The 15-day simulation from this method using Eq. (12). Also shown is the conservation of (bottom left) total energy (TE) and (bottom right) potential enstrophy (PE) from the three methods at grid level 6.
Citation: Monthly Weather Review 148, 10; 10.1175/MWR-D-20-0024.1
REFERENCES
Arakawa, A., and V. R. Lamb, 1977: Computational design of the basic dynamical processes of the UCLA general circulation model. Methods in Computational Physics, J. Chang, Ed., Vol. 17, Academic Press, 173–265.
Bell, M. J., P. S. Peixoto, and J. Thuburn, 2017: Numerical instabilities of vector-invariant momentum equations on rectangular C-grids. Quart. J. Roy. Meteor. Soc., 143, 563–581, https://doi.org/10.1002/qj.2950.
Bleck, R., and Coauthors, 2015: A vertically flow-following icosahedral grid model for medium-range and seasonal prediction. Part I: Model description. Mon. Wea. Rev., 143, 2386–2403, https://doi.org/10.1175/MWR-D-14-00300.1.
Choi, S.-J., and S.-Y. Hong, 2016: A global non-hydrostatic dynamical core using the spectral element method on a cubed-sphere grid. Asia-Pac. J. Atmos. Sci., 52, 291–307, https://doi.org/10.1007/s13143-016-0005-0.
Cullen, M. J. P., 1974: Integrations of the primitive equations on a sphere using the finite element method. Quart. J. Roy. Meteor. Soc., 100, 555–562, https://doi.org/10.1002/qj.49710042605.
Du, Q., V. Faber, and M. Gunzburger, 1999: Centroidal Voronoi Tessellations: Applications and algorithms. SIAM Rev., 41, 637–676, https://doi.org/10.1137/S0036144599352836.
Gassmann, A., 2013: A global hexagonal C-grid non-hydrostatic dynamical core (ICON-IAP) designed for energetic consistency. Quart. J. Roy. Meteor. Soc., 139, 152–175, https://doi.org/10.1002/qj.1960.
Gassmann, A., 2018: Discretization of generalized Coriolis and friction terms on the deformed hexagonal C-grid. Quart. J. Roy. Meteor. Soc., 144, 2038–2053, https://doi.org/10.1002/qj.3294.
Gates, W. L., E. S. Batten, A. B. Kahle, and A. B. Nelson, 1971: A documentation of the Mintz-Arakawa two-level atmospheric general circulation model. Rand Corp. Tech. Rep. R-877-ARPA, 408 pp., https://www.rand.org/pubs/reports/R0877.html.
Giraldo, F., J. Kelly, and E. Constantinescu, 2013: Implicit-explicit formulations of a three-dimensional Nonhydrostatic Unified Model of the Atmosphere (NUMA). SIAM J. Sci. Comput., 35, B1162–B1194, https://doi.org/10.1137/120876034.
Govett, M., L. Hart, T. Henderson, J. Middlecoff, and D. Schaffer, 2003: The scalable modeling system: Directive-based code parallelization for distributed and shared memory computers. Parallel Comput., 29, 995–1020, https://doi.org/10.1016/S0167-8191(03)00084-X.
Haurwitz, B., 1940: The motion of atmospheric disturbances on the spherical earth. J. Mar. Res., 3, 254–267.
Heikes, R., and D. A. Randall, 1995a: Numerical integration of the shallow-water equations on a twisted icosahedral grid. Part I: Basic design and results of tests. Mon. Wea. Rev., 123, 1862–1880, https://doi.org/10.1175/1520-0493(1995)123<1862:NIOTSW>2.0.CO;2.
Heikes, R., and D. A. Randall, 1995b: Numerical integration of the shallow-water equations on a twisted icosahedral grid. Part II: A detailed description of the grid and an analysis of numerical accuracy. Mon. Wea. Rev., 123, 1881–1887, https://doi.org/10.1175/1520-0493(1995)123<1881:NIOTSW>2.0.CO;2.
Hollingsworth, A., P. Kållberg, V. Renner, and D. M. Burridge, 1983: An internal symmetric computational instability. Quart. J. Roy. Meteor. Soc., 109, 417–428, https://doi.org/10.1002/qj.49710946012.
Jakob, R., J. J. Hack, and D. L. Williamson, 1993: Solutions to the shallow water test set using the spectral transform method. NCAR Tech. Note NCAR/TN-343+STR, 82 pp.
Lauritzen, P., C. Jablonowski, M. Taylor, and R. Nair, Eds., 2011: Numerical Techniques for Global Atmospheric Models. Lecture Notes in Computational Science and Engineering, Vol. 80, Springer, 564 pp.
Lee, J.-L., and A. E. MacDonald, 2009: A finite-volume icosahedral shallow-water model on a local coordinate. Mon. Wea. Rev., 137, 1422–1437, https://doi.org/10.1175/2008MWR2639.1.
Lin, S.-J., 2004: A “vertically Lagrangian” finite-volume dynamical core for global models. Mon. Wea. Rev., 132, 2293–2307, https://doi.org/10.1175/1520-0493(2004)132<2293:AVLFDC>2.0.CO;2.
Machenhauer, B., 1979: The spectral method. Numerical Methods Used in Atmospheric Models, GARP Publication Series, Vol. 17, World Meteorological Organization, 121–275.
Maday, Y., and A. T. Patera, 1989: Spectral element methods for the incompressible Navier-Stokes equations. State-of-the-Art Surveys on Computational Mechanics, A. K. Noor, Ed., American Society of Mechanical Engineers, 71–143.
Mandelbrot, B. B., 1982: The Fractal Geometry of Nature. Henry Holt and Company, 468 pp.
Miura, H., 2019: Application of the synchronized B-grid staggering for solution of the shallow-water equations on the spherical icosahedral grid. Mon. Wea. Rev., 147, 2485–2509, https://doi.org/10.1175/MWR-D-18-0304.1.
Miura, H., and M. Kimoto, 2005: A comparison of grid quality of optimized spherical hexagonal–pentagonal geodesic grids. Mon. Wea. Rev., 133, 2817–2833, https://doi.org/10.1175/MWR2991.1.
O’Neil, M., F. Woolfe, and V. Rokhlin, 2010: An algorithm for the rapid evaluation of special function transforms. Appl. Comput. Harmon. Anal., 28, 203–226, https://doi.org/10.1016/j.acha.2009.08.005.
Peixoto, P. S., 2016: Accuracy analysis of mimetic finite volume operators on geodesic grids and a consistent alternative. J. Comput. Phys., 310, 127–160, https://doi.org/10.1016/j.jcp.2015.12.058.
Peixoto, P. S., and S. R. Barros, 2014: On vector field reconstructions for semi-Lagrangian transport methods on geodesic staggered grids. J. Comput. Phys., 273, 185–211, https://doi.org/10.1016/j.jcp.2014.04.043.
Peixoto, P. S., J. Thuburn, and M. J. Bell, 2018: Numerical instabilities of spherical shallow-water models considering small equivalent depths. Quart. J. Roy. Meteor. Soc., 144, 156–171, https://doi.org/10.1002/qj.3191.
Perot, B., 2000: Conservation properties of unstructured staggered mesh schemes. J. Comput. Phys., 159, 58–89, https://doi.org/10.1006/jcph.2000.6424.
Phillips, N. A., 1959: Numerical integration of the primitive equations on the hemisphere. Mon. Wea. Rev., 87, 333–345, https://doi.org/10.1175/1520-0493(1959)087<0333:NIOTPE>2.0.CO;2.
Phillips, N. A., 1973: Principles of large scale numerical weather prediction. Dynamic Meteorology, P. Morel, Ed., Springer, 1–96.
Press, W. H., B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling, 1992: Numerical Recipes in Fortran 90: The Art of Scientific Computing. 2nd ed. Cambridge University Press, 963 pp.
Randall, D. A., 1994: Geostrophic adjustment and the finite-difference shallow-water equations. Mon. Wea. Rev., 122, 1371–1377, https://doi.org/10.1175/1520-0493(1994)122<1371:GAATFD>2.0.CO;2.
Rauscher, S. A., and T. D. Ringler, 2014: Impact of variable-resolution meshes on midlatitude baroclinic eddies using CAM-MPAS-A. Mon. Wea. Rev., 142, 4256–4268, https://doi.org/10.1175/MWR-D-13-00366.1.
Ringler, T. D., and D. A. Randall, 2002: The ZM grid: An alternative to the Z grid. Mon. Wea. Rev., 130, 1411–1422, https://doi.org/10.1175/1520-0493(2002)130<1411:TZGAAT>2.0.CO;2.
Ringler, T. D., J. Thuburn, J. B. Klemp, and W. C. Skamarock, 2010: A unified approach to energy conservation and potential vorticity dynamics for arbitrarily-structured C-grids. J. Comput. Phys., 229, 3065–3090, https://doi.org/10.1016/j.jcp.2009.12.007.
Sadourny, R., and C. Basdevant, 1985: Parameterization of subgrid scale barotropic and baroclinic eddies in quasi-geostrophic models: Anticipated potential vorticity method. J. Atmos. Sci., 42, 1353–1363, https://doi.org/10.1175/1520-0469(1985)042<1353:POSSBA>2.0.CO;2.
Sadourny, R., A. Arakawa, and Y. Mintz, 1968: Integration of the nondivergent barotropic vorticity equation with an icosahedral-hexagonal grid for the sphere. Mon. Wea. Rev., 96, 351–356, https://doi.org/10.1175/1520-0493(1968)096<0351:IOTNBV>2.0.CO;2.
Satoh, M., T. Matsuno, H. Tomita, H. Miura, T. Nasuno, and S. Iga, 2008: Nonhydrostatic ICosahedral Atmospheric Model (NICAM) for global cloud resolving simulations. J. Comput. Phys., 227, 3486–3514, https://doi.org/10.1016/j.jcp.2007.02.006.
Sela, J. G., 1980: Spectral modeling at the National Meteorological Center. Mon. Wea. Rev., 108, 1279–1292, https://doi.org/10.1175/1520-0493(1980)108<1279:SMATNM>2.0.CO;2.
Simmons, A. J., D. M. Burridge, M. Jarraud, C. Girard, and W. Wergen, 1989: The ECMWF medium-range prediction models development of the numerical formulations and the impact of increased resolution. Meteor. Atmos. Phys., 40, 28–60, https://doi.org/10.1007/BF01027467.
Skamarock, W. C., J. B. Klemp, M. G. Duda, L. D. Fowler, S.-H. Park, and T. D. Ringler, 2012: A multiscale nonhydrostatic atmospheric model using centroidal Voronoi tesselations and C-grid staggering. Mon. Wea. Rev., 140, 3090–3105, https://doi.org/10.1175/MWR-D-11-00215.1.
Stuhne, G., and W. Peltier, 1996: Vortex erosion and amalgamation in a new model of large scale flow on the sphere. J. Comput. Phys., 128, 58–81, https://doi.org/10.1006/jcph.1996.0196.
Stuhne, G., and W. Peltier, 1999: New icosahedral grid-point discretizations of the shallow water equations on the sphere. J. Comput. Phys., 148, 23–58, https://doi.org/10.1006/jcph.1998.6119.
Taylor, M., J. Tribbia, and M. Iskandarani, 1997: The spectral element method for the shallow water equations on the sphere. J. Comput. Phys., 130, 92–108, https://doi.org/10.1006/jcph.1996.5554.
Thuburn, J., and Y. Li, 2000: Numerical simulations of Rossby-Haurwitz waves. Tellus, 52A, 181–189, https://doi.org/10.3402/tellusa.v52i2.12258.
Thuburn, J., T. D. Ringler, W. C. Skamarock, and J. B. Klemp, 2009: Numerical representation of geostrophic modes on arbitrarily structured C-grids. J. Comput. Phys., 228, 8321–8335, https://doi.org/10.1016/j.jcp.2009.08.006.
Tomita, H., M. Tsugawa, M. Satoh, and K. Goto, 2001: Shallow water model on a modified icosahedral geodesic grid by using spring dynamics. J. Comput. Phys., 174, 579–613, https://doi.org/10.1006/jcph.2001.6897.
Tomita, H., M. Satoh, and K. Goto, 2002: An optimization of the icosahedral grid modified by spring dynamics. J. Comput. Phys., 183, 307–331, https://doi.org/10.1006/jcph.2002.7193.
Vestine, E. H., W. L. Sibley, J. W. Kern, and J. L. Carlstedt, 1963: Integral and spherical-harmonic analyses of the geomagnetic field for 1955.0, Part II. J. Geomag. Geoelectr., 15, 73–89, https://doi.org/10.5636/jgg.15.73.
Voosen, P., 2018: Science insurgents plot a climate model driven by artificial intelligence. Science, 361, 344–347, https://doi.org/10.1126/science.aau8974.
Wan, H., and Coauthors, 2013: The ICON-1.2 hydrostatic atmospheric dynamical core on triangular grids — Part I: Formulation and performance of the baseline version. Geosci. Model Dev., 6, 59–119, https://doi.org/10.5194/gmdd-6-59-2013.
Wang, N., and J. Lee, 2011: Geometric properties of the icosahedral-hexagonal grid on the two-sphere. SIAM J. Sci. Comput., 33, 2536–2559, https://doi.org/10.1137/090761355.
Wedi, N. P., M. Hamrud, and G. Mozdzynski, 2013: A fast spherical harmonics transform for global NWP and climate models. Mon. Wea. Rev., 141, 3450–3461, https://doi.org/10.1175/MWR-D-13-00016.1.
Williamson, D. L., 1968: Integration of the barotropic vorticity equation on a spherical geodesic grid. Tellus, 20, 642–653, https://doi.org/10.3402/tellusa.v20i4.10044.
Williamson, D. L., 2007: The evolution of dynamical cores for global atmospheric models. J. Meteor. Soc. Japan, 85B, 241–269, https://doi.org/10.2151/JMSJ.85B.241.
Williamson, D. L., J. B. Drake, J. J. Hack, R. Jakob, and P. N. Swarztrauber, 1992: A standard test set for numerical approximations to the shallow water equations in spherical geometry. J. Comput. Phys., 102, 211–224, https://doi.org/10.1016/S0021-9991(05)80016-6.
Winninghoff, F. J., 1968: On the adjustment towards a geostrophic balance in a simple primitive equation model with application to the problems and objective analysis. Ph.D. thesis, University of California, 161 pp.
World Meteorological Organization, 1969: Lectures on Numerical Short-Range Weather Prediction: World Meteorological Organization Regional Training Seminar. Hydrometeoizdat, 706 pp.
Xie, Y., 2019: Generalized Z-grid model for numerical weather prediction. Atmosphere, 10, 179, https://doi.org/10.3390/atmos10040179.
Zängl, G., D. Reinert, P. Rípodas, and M. Baldauf, 2015: The ICON (ICOsahedral Non-hydrostatic) modelling framework of DWD and MPI-M: Description of the non-hydrostatic dynamical core. Quart. J. Roy. Meteor. Soc., 141, 563–579, https://doi.org/10.1002/qj.2378.
