1. Introduction
The spatial resolution of atmospheric models has been recently increasing to explicitly treat cloud dynamics using microphysics schemes. In the Dynamics of the Atmospheric general circulation Modeled On Non-hydrostatic Domains (DYAMOND) project, Stevens et al. (2019) compared the results of various global high-resolution models with a several-kilometer mesh in the horizontal direction: the Non-hydrostatic ICosahedral Atmospheric Model (NICAM; Tomita and Satoh 2004; Satoh et al. 2014), the ICOsahedral Non-hydrostatic model (ICON; Zängl et al. 2015), the Model for Prediction Across Scales (MPAS; Skamarock et al. 2012), and the Goddard Earth Observing System model version 5 (GEOS-5; Putman and Suarez 2011). As an extreme high-resolution case, Miyamoto et al. (2013) successfully conducted global cloud-resolving simulation with a subkilometer horizontal mesh. Considering atmospheric turbulence, the spatial resolutions of those global simulations with several- or subkilometer horizontal mesh are still insufficient for the explicit representation of boundary layer turbulent eddies. Usually, the effect of turbulence in such simulations is often parameterized by Reynolds-averaged Navier–Stokes (RANS) models (e.g., Mellor and Yamada 1974).
In contrast, many studies (e.g., Wyngaard 2004; Honnert et al. 2020) have pointed out that the RANS model is inappropriate for O(10–100) m resolutions; it doubly counts the turbulent eddy effect because large-scale eddies in the planetary boundary layer (PBL) begin to be resolved. Wyngaard (2004) named this resolution regime the “terra-incognita,” and the concept is well known as the “gray zone problem” of turbulence (Honnert et al. 2020). To avoid this problem, the turbulence scheme should be changed to that of large-eddy simulation (LES), whose main concept is that only unresolved small-scale eddies are parameterized. The Smagorinsky–Lilly turbulence scheme (Smagorinsky 1963; Lilly 1962) is one such model, although turbulent parameterizations similar to the Smagorinsky scheme are sometimes used in general circulation models (e.g., Becker and Burkhardt 2007). To determine the characteristics of shallow clouds, LES is employed for meteorological simulations using regional models (e.g., Sommeria 1976; Savic-Jovcic and Stevens 2008; Corbetta et al. 2015; Sato et al. 2015). Heinze et al. (2017) conducted LES over a wide domain covering Germany. Stevens et al. (2020) discussed the added value of LES for simulating clouds by performing LES over a wide domain. Although LES in the global domain is still challenging owing to the limitations of computational resources, these studies indicate a link of the global LES. Recent computer technology developments will enable us to perform these simulations with a horizontal resolution of O(10–100) m in the near future (e.g., Satoh et al. 2019).
As is well recognized, LES requires a high numerical accuracy for spatial discretization. Nevertheless, state-of-the-art global models employ relatively low-order of spatial accuracy, such as second-order accuracy (e.g., Tomita and Satoh 2004; Zängl et al. 2015). In contrast, higher spatial accuracy is often used in the regional models (e.g., Skamarock et al. 2008; Nishizawa et al. 2015), although such high-order accuracy is applied only to the advection terms and is guaranteed only for uniform flow. Historically, the numerical accuracy of dynamical cores has not always been the primary factor in the numerical performance of atmospheric models because of the uncertainties of physical processes. However, the low-order discretization becomes critical to precisely perform LES for the following reasons.
Consider a situation in which the numerical error due to low-order advection schemes is significant or explicitly added numerical filters clearly affect the flow field. Basically, such errors can contaminate the subgrid-scale (SGS) eddy viscosity terms (e.g., Rogallo and Moin 1984). Vreman et al. (1994) evaluated the numerical error terms and SGS terms, noting that if the filter length of LES is equal to the grid spacing, the spatial discretization errors are more significant than the SGS terms even for the fourth-order spatial scheme. Because the low-order scheme requires greater filter length and increases computational cost, they recommended higher-order schemes. Ghosal (1996) formulated the numerical error of the finite-difference method (FDM). He showed that even if higher-order FDM is employed for nonlinear terms, the aliasing error contaminates the solutions. In addition to the suggestion of Vreman et al. (1994) for the filter length, Ghosal (1996) argued that numerical filters should be adopted to selectively suppress flow structures smaller than the filter length. He also showed that, if eighth-order FDM is employed with a filter length of double grid spacing, the spatial error against SGS terms decreases by two orders of magnitude. In contrast, Brandt (2006) noted that if the filter length is too long, the SGS terms are underestimated because the structural information corresponding to the energy containing wavelength is lost. Brandt (2006) proposed that the spatial filtering only for advection terms improved the results.
In this manner, the problem of dominating the numerical error terms over the SGS terms has been of historical concern in the research field of computational fluid dynamics (CFD). We should consider a further complicated situation in meteorological simulations: e.g., the stratification, vertical shear, and a wide dynamic range of wind velocity. There is still room to theoretically reconsider this problem, and confirm it using actual numerical experiments. Therefore, it is important to extend the knowledge of the CFD area to meteorology, paying attention to the aforementioned complicated factors. In this study, one of the factors considered is the effect of stratification, as discussed in section 4a. At the same time, to obtain clearer requirements of the numerical accuracy for dynamical cores, we should also consider the temporal discretization error that previous works have not addressed.
In this study, we derived two criteria for the order of the numerical accuracy of advection terms, in which the numerical error terms do not dominate over the SGS terms, focusing on the LES of atmospheric boundary layer turbulence. Our work follows the theory of three-dimensional homogeneous isotropic turbulence, paying attention to a situation peculiar to meteorology. In addition, our approach is based on a more intuitive idea than that used in previous studies. We split the SGS terms into the diffusive and advective contributions, and compared them with numerical diffusion and dispersion accompanied by the discretization of advection terms, respectively. We investigated whether the derived criteria are useful for actual simulations. Finally, including the influence of temporal discretization, we clarified how high the order of numerical accuracy must be in space and time.
In the next section, we derive two ratios associated with numerical diffusion and dispersion. They provide the criteria for which the numerical errors do not dominate over the SGS terms and explain the qualitative behavior for the order of accuracy. As a typical problem, in section 3, we perform ideal numerical experiments of PBL turbulence using advection schemes with various order accuracy, investigating the influence of numerical accuracy on statistical quantities of turbulence. In section 4, we discuss the total numerical error, including temporal discretization error and spatial discretization error with SGS terms. Section 5 provides our conclusion together with scope for future work.
2. Theoretical formulation of criteria for numerical accuracy
a. Numerical error due to advection scheme
Numerical error terms for approximating the first derivative term using the finite-difference scheme. Here, fx(n) = ∂nf/∂xn.
Actually, the temporal discretization also produces the numerical diffusion and dispersion errors. Here, we consider that such temporal errors are ideally small if an appropriate high-order scheme is used in time. We discuss its validity in section 4 in detail.
b. Criterion due to numerical diffusion
c. Criterion due to numerical dispersion
d. Behavior of derived criteria
How do Rdiff and Rdisp behave with decreasing grid spacing? This can be considered one of the main objectives of this study. Both of the two ratios should be sufficiently smaller than unity for the SGS terms to be effective. Here, we theoretically discuss this issue, considering two ratios as functions of grid spacing and wavelength. For this purpose, three definitions are introduced as follows.
3. Turbulence simulation in an idealized planetary boundary layer turbulence by LES
In this section, to verify the numerical criteria derived in the previous section, we performed a series of numerical experiments for atmospheric turbulence in an idealized PBL. Although the numerical criteria were formulated under the Boussinesq approximation, here we used an LES model based on compressible equations. Such equations are often adopted in modern meteorological models to avoid the difficulties associated with the approximation of compressibility (e.g., solving the pressure Poisson equation in massive parallel computational environments). In contrast, based on Erlebacher et al. (1992), the compressibility effect is considered to have negligible impact on the turbulent fields discussed in this study.
a. Model and experimental setup
The experimental setup is based on Nishizawa et al. (2015, hereafter N2015). The model used is SCALE-RM (N2015; Sato et al. 2015). The computational domain has a square area of 9.6 km × 9.6 km with a double periodic boundary condition and an altitude of 3 km. In the dynamical process, the fully compressible nonhydrostatic equations are solved. The turbulent process is expressed by the Smagorinsky–Lilly model considering the stratification effect (Brown et al. 1994). The filter length is set to two times the grid spacing. To focus on the turbulent process, the radiation and moist processes are absent. For the initial atmospheric condition, we provide the potential temperature that increases with height at a rate of 4 K km−1 and add the random disturbance with an amplitude of 1 K. The initial wind in the x direction is 5 m s−1. At the surface, a constant heat flux of 200 W m−2 is continuously injected.
The numerical method is summarized as follows. The governing equations were spatially discretized by a conservative FDM with the Arakawa-C type staggered grid (Arakawa and Lamb 1977). For the advection terms with flux form, the spatial discretization was selected among the second, fourth, sixth, and eighth-order central, and the third-, fifth-, and seventh-order upwind schemes. Hereinafter, the n-order central and upwind schemes are denoted as CDn and UDn, respectively. The pressure gradient and SGS terms are spatially discretized by CD2. When using CD schemes for the advection terms, numerical diffusion is explicitly added for the numerical stabilization. We prepared the second-, fourth, sixth-, and eighth-order differential operators, denoted by NDn. To prevent the reflection of waves at the upper boundary, we placed a sponge layer above 2 km. The horizontal grid-spacing was 10 m and the vertical grid spacing was the same as that in the horizontal direction except the sponge layer. As for the temporal discretization, a fully explicit scheme was applied to all terms. We chose an explicit Runge–Kutta (RK) scheme among the three-stage and third-order (Wicker and Skamarock 2002), the classical four-stage and fourth-order, the seven-stage and sixth-order (Lawson 1967), and the 11-stage and eighth-order (Cooper and Verner 1972) schemes. The n-order RK scheme is denoted by RKn. Although the straightforward choice for the verification of our criteria would be to adopt a temporal scheme whose order of accuracy is higher than that of the highest-order advection scheme, such a choice is too difficult in terms of computational costs. Thus, we required that the order of temporal accuracy be at least the same as that of each advection scheme to avoid the situation where the temporal errors significantly dominate the numerical errors due to advection. The time step for the dynamical process was 0.012 s, and the SGS terms were evaluated at every stage of the RK scheme. For the central schemes, the explicit numerical diffusion was evaluated in the final stage of the RK scheme.
Table 2 provides a summary of the experiments. We performed eight experiments with a spatial resolution of 10 m, changing the advection scheme among the various order accuracies. CD8ND8 has the highest numerical accuracy in space and time and is referred to as the “control experiment.” The differential order of explicit numerical diffusion is basically equal to the order of accuracy of the CD scheme. For comparison with N2015, we used higher-order numerical diffusion only in CD4ND8. The intensity of the explicit numerical diffusion was set to γ2n = 2.0 × 10−4 in Eq. (6); the value is the same as N2015. In this setup, γadv is approximately 3.3 × 10−2, which is less than the corresponding values for UD schemes. In CD2ND2 and CD8ND8, we also performed experiments with grid spacings of 40 or 80 m in which the time step increased such that the Courant number was fixed. Furthermore, to discuss the influence of temporal errors, we performed two additional experiments in which RK8 in CD8ND8 was replaced with RK3 and RK4. For the control experiment, the temporal integration was performed for 4 h. For the other cases with 10 m resolution, to reduce computational costs, it was performed for only 1 h by setting the output at 3 h from the control experiment as the initial condition. The results for the last 30 min were analyzed.
Summary of a series of the numerical experiments for atmospheric turbulence in an idealized PBL.
In our computational configuration,1 computational and communication costs due to the higher-order advection scheme were negligible, whereas the increase in the overall computational time was caused by the increase in the stage of the high-order RK scheme. Temporal integration for 4 h for the control experiment required approximately 12 days. The reason for the relatively low computational cost of the high-order advection scheme is a virtue of the structured grid, such as the continuous or striped access of the memory space and effective use of the cache memory. These benefits are not necessarily guaranteed for an unstructured grid used in several recent global nonhydrostatic models. In addition, if MPI processes are increased by O(104) with a reduction in the number of grids per process, communication costs with an increase of halo in high-order schemes will become nonnegligible. Thus, it is possible to underestimate the computational costs of high-order advection schemes, although we have not focused on this issue in the present study.
b. Results of the control experiment
After 4 h, the top of the PBL reaches approximately 1200 m in altitude. Figure 1a shows the horizontal distribution of the vertical velocity at z = 500 m as a typical field in the middle of PBL. The convective cells have well-known hexagonal or quadrangular structures. The updraft within the narrow boundaries of the cells is stronger than the downdraft in the remaining region. These features of the flow fields are qualitatively the same as those shown in Fig. 3a of N2015.
From the control experiment that provides the highest accuracy, we attempted to estimate the unknown parameter η as follows. We generated a series of coarsened data using the top-hat filter, whose window size D increased from 30 to 350 m. For each D, |S| was evaluated using a tenth-order central difference on the original mesh, and ∇|S| was evaluated by subsequently differentiating |S|. Figure 1b shows the distribution of |S| for D = 70 m, as an example. Figure 2 shows the dependence of the square root of horizontally averaged |S|2 and (∇|S|)2 on D, with Eqs. (17) and (18) for three values of η. These slopes are consistent with the values expected by the Kolmogorov theory: −2/3 and −5/3 for |S| and∇|S|, respectively. From Fig. 2, we can estimate the value of η as 0.15, although it varied by approximately 20%. Ideally, η should be considered a constant if the energy spectra precisely obey the −5/3 power law with the same energy dissipation rate, as derived in appendix B. This constancy of η is also indicated by coarsened data based on CD8ND8 experiments with grid spacings of 40 and 80 m (refer to the green and blue lines in Fig. 2). The reason for the 20% variation is that the actual energy spectra did not precisely obey the −5/3 power law over the entire wavelength range. In addition, the filter used to obtain the coarsened data is considered to affect this behavior.
The typical wind speed is also necessary to quantify our numerical criteria defined by Eqs. (19) and (20). Figure 3 shows the histograms of wind velocity at z = 500 m. The maximum wind frequencies in the x and y directions are approximately 5 and 0 m s−1, respectively. In contrast, the maximum frequency of the vertical wind is negative and at approximately −0.5 m s−1.
The numerical diffusion and dispersion should be smaller than the first term (Laplacian term) and the second term (dispersion term) in Eq. (4), respectively. However, if the typical sizes of the two terms in Eq. (4) are considerably different, we should not consider that the importance of Rdiff is comparable with that of Rdisp. To examine the actual contributions, Fig. 4 shows the histogram of the ratios of the dispersion term to the Laplacian term at each grid point. Although the Laplacian terms usually dominate the dispersion terms, in 15% grid points, the dispersion terms are comparable to or larger than the Laplacian terms. Thus, we consider the importance of Rdiff and Rdisp to be equal.
c. Quantification of numerical criteria
We discuss here the analytical dependence of Rdiff in Eq. (19) and Rdisp in Eq. (20) on the grid spacing and the wavelength, using the determined parameters: m = 2 in Eq. (15), η = 0.15 in Eq. (17), |U| = 5 m s−1, and γadv = 3.3 × 10−2 for the CD schemes. The values of Rdiff and Rdisp at which the numerical errors are considered acceptable are arbitrary. Here, however, we set the criteria in which they should be below 10−1 for more than eight grids: the one-order small value may be appropriate for practical cases, and targeting flow structures longer than eight grids is appropriate in terms of effective resolution, according to Skamarock (2004) and Abdalla et al. (2013).
The left panel in Fig. 5 and Fig. 5b show the results for CD schemes. In CD2ND2, Rdiff is no longer smaller than 10−1 at the grid spacing usually used in LES. For CD4ND4, Rdiff is approximately 10−1 around l ~ 10 in the O(10) m resolution. The order is the boundary of whether our criterion of numerical diffusion is satisfied. For higher-order numerical diffusion than CD4ND4, this criterion is satisfied. The constraint from the numerical dispersion is more severe. Tighter constraints are imposed on the order accuracy of numerical schemes. For example, even for CD8ND8, the criterion of Rdisp < 10−1 is not satisfied unless the structure is longer than eight grids.
The right panel in Fig. 5 (except for Fig. 5b) shows the results for UD schemes. Rdiff for the (2n − 1)-order UD scheme is two orders of magnitude larger than that for the 2n-order CD scheme. In the O(10) m resolution, UD7 satisfy the criterion of numerical diffusion but UD3 and UD5 are not acceptable. The constraint of numerical dispersion with the (2n − 1)-order UD scheme is the same as that with the 2n-order CD scheme. In UD3 and UD5 schemes, the constraint of numerical diffusion is more severe than that of numerical dispersion at the wavelength longer than eight grids.
Here, we provide a short summary of the spatial accuracy of advection terms with a focus on the flow structure longer than eight grids. In terms of numerical diffusion, at least a fourth-order accuracy is necessary for CD schemes, provided that the differential order of explicit numerical diffusion terms is the same or higher than the order of the used scheme. At that time, the intensity of explicit numerical diffusion needs to be one or two order less than that of the third-order upwind scheme. The UD scheme requires at least seventh-order accuracy to satisfy the criterion. In terms of numerical dispersion, at least a seventh-order scheme is necessary in both the CD and UD schemes.
d. Validation of numerical criteria
In this section, we will verify the criteria quantified in section 3c. From the viewpoint of energy spectra, we focus on the difference between the advection schemes in Table 2 while referring to Fig. 5.
Figure 6 shows the density-weighted energy spectra of the three-dimensional velocity at z = 500 m. Regardless of the schemes, the energy spectra obey the −5/3 power law in the wavelength range of 103–102 m, and they drop with a steeper slope at wavelengths shorter than 10 grids. The energy spectra for CD4ND8 well reproduce those of the control experiment shown in Fig. 1 of N2015.
The implication of Rdiff depicted by Fig. 5 is consistent with the energy spectra. The most diffusive one is CD2ND2 followed by UD3. When comparing the two results, Rdiff for CD2ND2 is slightly smaller than that for UD3 at a shorter wavelength (less than approximately 30 grids); however, it is larger than that for UD3 at longer wavelengths. Figure 7 also indicates that the two schemes do not capture the smaller-scale structure of flow near the updraft region of convections well, compared to other schemes. Among the CD schemes, the energy spectra essentially depend on the order of the differential operator of the numerical diffusion, in particular, over short wavelengths. For the CD schemes with more than fourth-order numerical diffusion, the energy spectra obey the −5/3 power law more at wavelengths shorter than eight grids. For higher-order UD schemes, Rdiff for UD5 is larger than for CD4ND4 for wavelengths shorter than approximately 10 grids; for UD7, it has an intermediate value between CD4ND4 and CD6ND6 for the wavelength between approximately 6 grids and 20 grids. As suggested in Fig. 5, the energy spectra for UD5 and UD7 schemes are actually more diffusive than those for CD4ND4 and CD6ND6, respectively.
Although it is difficult to directly estimate the influence of Rdisp, it can be indirectly inferred from the above energy spectra. Here, we compare schemes that have the same numerical diffusion terms but different numerical dispersion terms. We focus on the difference between CD4ND8 and CD8ND8 and that between UD7 and CD8ND8. Figure 6 shows that the former is smaller than the latter. This implies that the influence of numerical dispersion on the energy spectra tends to be smaller than that of numerical diffusion. Nevertheless, note that it is possible to underestimate the problems of the numerical dispersion error because the current experiment is based on an idealized setup. The problem may appear essentially under more realistic situations.
Figure 5 indicates that the influence of the numerical diffusion in comparison to the SGS term strengthens as the grid spacing decreases. To examine this behavior in numerical experiments, Fig. 8 shows the density-weighted energy spectra obtained from CD2ND2 and CD8ND8 with different grid spacings. Here, we focused on the wavelength at which the energy spectra of CD2ND2 begin to deviate from that of CD8ND8. This wavelength was approximately 10-grids for a grid spacing of 80 m, whereas it was approximately 30 grids for a grid spacing of 10 m. Thus, the range of wavelengths for which the numerical diffusion errors significantly affected the energy spectra became wider as the grid spacing decreased.
4. Discussion
a. Effect of stratification on eddy viscosity
In section 2, we assume f(Ri) = 1 in Eq. (3). However, in our numerical experiments in section 3, where we use the turbulent model based on Brown et al. (1994), f(Ri) is modified by the nonneutral stratification effect as follows:
To discuss the influence of stratification on Rdiff, Fig. 9a shows the histogram of the Richardson number Ri at z = 500 m and the dependence on the resolution. Here, using the results of the control experiment, we calculated Ri for the different resolutions in same manner in which |S| was obtained in section 3b. At almost all the grid points, the magnitude spreads in the range of |Ri| < 10−1. As the resolution increases, the influence of stratification on Ri tends to decrease, because the peak value increases at Ri ~ 0. Figure 9b shows the histogram of f(Ri) for various coarsened data. For the unstable case, f(Ri) has a value between 1 and 2. Thus, unstable stratification only affects Rdiff by a factor of 2 at most. For weakly stable stratification case, at approximately 10% grid points, f(Ri) is less than 10−1.
To investigate the influence of stratification on Rdisp, Fig. 10 shows the histogram of the ratio ∂f(Ri)|S|/∂x to ∂|S|/∂x; this ratio is between 10−1 and 101 at approximately 90% grid points.
These facts indicate that the criteria with numerical errors derived for neutral stratification in section 2 can be approximately adopted for most grid points in our numerical experiments. For stable stratification case, it is more probable that the SGS terms become significantly smaller than the numerical errors. Such a situation occurs at only 10% grid points where the required order of accuracy indicated by Rdiff and Rdisp for the neutral stratification is not sufficient.
b. Influence of temporal numerical errors on Rdiff and Rdisp
In section 2, the temporal discretization is not treated. To investigate the influence of the temporal error terms on the numerical criteria, let us consider the one-dimensional linear advection equation discretized by CD8ND8 in space and RKn schemes in time (n = 8, 4, 3).
To verify the above consideration in actual simulations, we performed two additional experiments, in which the RK8 of the CD8ND8 experiment was replaced by RK4 and RK3. Figure 11 shows that the energy spectra are consistent with the above consideration. The difference between RK8 and RK3 is considerably smaller than that in the advection schemes. The temporal accuracy is not as essential as the spatial accuracy for advection terms. This is because cr ≪ 1 when the sound wave is solved using a fully explicit method.
If otherwise we treat fast waves with semi-implicit temporal schemes or apply temporal splitting methods, the time step allows for cr ~ 1. In such cases, the temporal error affects Rdiff and Rdisp more. Therefore, the temporal discretization must be as accurate as spatial discretization.
In section 2, we derive the numerical criteria Rdiff and Rdisp, based on the modified equation of one-dimensional linear advection equation in FDM. However, if we precisely consider the corresponding modified equation for the multidimensional case, the cross terms as ∂2ϕ/∂x∂y appear (e.g., Durran 2010), depending on numerical schemes. Because the leading temporal errors often include these cross terms, we need to take care of the additional contributions, in particular for the case of cr ~ 1.
In our numerical experiments, the tendency due to the turbulent process was calculated at each RK stage for the dynamical process. However, in practical atmospheric models, some of physics parameterizations are typically coupled with the dynamical process in a temporal splitting fashion. Such strategies often reduce the order of temporal accuracy to the first or even lower orders (e.g., Wan et al. 2020). We should also consider the temporal errors due to physics-dynamics coupling.
c. Influence of spatial numerical error with SGS terms on the numerical criteria
In section 2, we assume that the SGS terms have a sufficient order of accuracy. However, CD2 is often adopted for its discretization. Such a low-order scheme would degrade the overall accuracy.
If we adopt CD2 discretization for SGS terms, the e-folding decay time is modified as
However, we must consider that, as the accuracy of the advection scheme increases, the numerical error terms become smaller compared to the “true” SGS terms. Because Fig. 5g shows that Rdiff for CD8ND8 is smaller than 10−2 for l > 8, the explicit numerical diffusion in CD8ND8 increase the effective (eddy viscosity plus numerical diffusion) decay intensity by less than O(1%) between the 8 and 16 grids. In contrast, if we introduce
d. Design of numerical filters on high-order discretization
Even if we use higher-order schemes, the numerical error inevitably affects the structure of several grid scale. Furthermore, when they are adopted for nonlinear terms, we must consider aliasing errors, as argued in Ghosal (1996). These facts are important for the coupling between dynamical and physical processes because the physical parameterizations often assume that the numerical errors in the dynamical process are small. A remedy for this problem is to properly coarsen the spatial distributions of variables treated in the dynamical process and give them to the physical processes. In this case, we should design an appropriate filter that effectively damps the structure between the two-grid scale and the effective resolution, in addition to the aliasing errors. Although this idea has already been verified in the context of LES (e.g., Vreman et al. 1994; Ghosal 1996), for other physical parameterizations in atmospheric models, the effect must be explored further.
5. Conclusions
In recent dynamical cores in state-of-the-art models, relatively low-order schemes are used for spatial discretization. However, as the resolution increases for LES, a problem arises in that the errors of such low-order schemes contaminate the SGS terms. In this study, we revealed the numerical accuracy of the advection schemes necessary for atmospheric LES. Two criteria were derived for this purpose, based on the theory of homogeneous isotropic three-dimensional turbulence. The first is the ratio of the e-folding time with the SGS terms to that with the numerical diffusion. The second is the ratio of the phase speed with the numerical dispersion to that with the SGS terms. We verified the criteria by performing an LES in a typical planetary boundary layer turbulence with various numerical orders of advection schemes.
In terms of numerical dissipation, the criterion suggests the following. For the third-order and fifth-order upwind schemes or second-order central scheme with second-order numerical diffusion, numerical diffusion significantly dominates the diffusion due to the eddy viscosity terms even at wavelengths longer than eight grids. Hence, a seventh-order accuracy is required for the upwind schemes in this case. In general, the upwind scheme has inevitably implicit diffusion. As it is determined by the local wind speed, we cannot control the degree of numerical diffusion explicitly. In contrast, for the central schemes, we can adjust the numerical diffusion under the condition of numerical stability. To rapidly remove gridscale noisy structures and properly work the eddy viscosity at an effective resolution, a higher differential operator than the fourth-order should be used. If a fourth-order numerical diffusion is used nevertheless, the nondimensional intensity of the numerical diffusion associated with the advection time scale should be set to one or two orders of magnitude smaller than that for the third-order upwind schemes. We confirmed that the above implication is consistent with the energy spectra obtained from the actual LES.
To satisfy the criterion from the numerical dispersion in the range of more than eight grids, seventh-order accuracy at least is required in the present experiment. For the central schemes, this constraint is more severe than that of high-order explicit numerical diffusion. Despite this strong constraint, the effect of the numerical dispersion error on the energy spectra is limited in our experiment, although it is not difficult to imagine that the numerical dispersion may affect the local turbulence mechanism. We must more intensively examine it in realistic experiments for future work.
We also examined the influence of temporal discretization on the criteria. In the atmospheric model used in this study, the time step is restricted by the sound wave, and the Courant number with advection is sufficiently smaller than unity. Thus, the temporal accuracy is not as essential as the spatial accuracy. For example, when an eighth-order spatial scheme is applied, eighth-order accuracy for the temporal scheme is excessive, and even the fourth-order temporal scheme is acceptable.
We statistically discuss the effect of stratification on our derived criteria in our numerical experiment. The numerical criteria for neutral stratification can be approximately applied for most grid points, because of the nearly neutral stratification. However, we must take care of approximately 10% grid points with Ri > 0.1, where a much higher order of accuracy is required than that for the neutral stratification.
In this paper, we focused on the advection terms that directly affect the turbulence simulations. Future studies should investigate other terms (e.g., the pressure gradient term including the isotropic part of the SGS strain tensor), and evaluate the impact of the discretization accuracy for such terms. Furthermore, considering atmospheric calculations with topography, we must evaluate the numerical errors associated with the vertical coordinate transformation. When adopting conventional high-order finite-volume schemes, we will face some difficulty in particular for the global models. The method is complex because of the distinction between the cell-center point and cell-averaged values, and the computational locality deteriorates as the stencil is extended. One promising method to counter these problems is the discontinuous Galerkin method (DGM). The performance of the DGM for dynamical cores has been evaluated (e.g., Giraldo and Restelli 2008; Kelly and Giraldo 2012; Marras et al. 2016). However, the practical performance in actual atmospheric simulations with physical processes has not been widely investigated. In terms of the balance between the physical representation and computational performance, investigating the suitability of DGM through the framework of this study should also be a focus of future work.
Acknowledgments
This research was supported by the JST AIP Grant JPMJCR19U2, Japan, and MEXT KAKENHI Grant JP20H05731. The experiments in this study were performed using the Oackbridge-CX supercomputer at the University of Tokyo and the supercomputer Fugaku at RIKEN (Project ID: ra000005 and hp200271). The authors are grateful to Team SCALE for providing the SCALE version 5.3.6. We thank Dr. Seiya Nishizawa and the reviewers for their valuable comments and suggestions. We thank Editage (www.editage.jp) for English-language editing.
Data availability statement
Source codes and configuration files for SCALE-RM used in this study are available from the Zenodo repository (http://doi.org/10.5281/zenodo.4915846). There, supplementary material helpful for the derivation of some equations in this paper is included. All data obtained from the numerical experiments has been deposited in the local storage at RIKEN R-CCS.
APPENDIX A
Numerical Diffusion and the Associated e-Folding Decay Time
a. General form of the finite-difference approximation of the first derivative
For equal grid spacing, we show the coefficient Ap in the finite-difference approximation of the first derivative term. Here, we assume that U > 0 in the upwind schemes.
b. Derivation of e-folding decay time: For CD schemes
c. Derivation of e-folding decay time: For UD schemes
Figures A1a and A1b shows that
APPENDIX B
Dependency of Typical Magnitude of Strain Tensor on the Resolution
APPENDIX C
Influence of Numerical Errors due to Temporal Discretization
In this section, we investigate the influence of temporal discretization errors using the one-dimensional linear advection equation of Eq. (A1).
a. Derivation of the modified equation
In our experiments, where the Courant number with advection cr is significantly less than 1, the effect of the higher-order temporal error terms is expected to be small. To verify it, we calculate the amplification factor from Eq. (C6). Figure C1a shows the factor assuming that the spatial error is much smaller than the temporal error. When cr ~ O(10−1), the effect of the higher-order temporal error terms than
b. Modification of the numerical criteria due to temporal errors
We could approximately consider the maximum Courant number associated with the advection such that the temporal errors are no longer acceptable. For example, let us consider where we adopt the RK4 scheme and CD8ND8 for advection terms and appropriately treat the terms associated with fast waves using a temporal strategy. Furthermore, we set the criteria with temporal errors in which the leading contributions of the temporal error terms in Eqs. (21) and (22) should be less than 10% at l ~ 102 for the O(10) m resolution. The typical magnitudes of the factors, except for cr, are still given by the values in Table 3. The criteria with temporal errors require that cr < 0.07, and cr < 0.04 for Rdiff and Rdisp, respectively. Thus, if we set the model time step as long as the numerical stability for advection is ensured, the criteria with temporal errors can possibly be violated. This indication is consistent with the discussion of large time step in section 4b.
Although we focus on some RK schemes in this study, in the same manner as that used here, we can evaluate the impact of temporal errors for other temporal schemes.
REFERENCES
Abdalla, S., L. Isaksen, P. Janssen, and N. Wedi, 2013: Effective spectral resolution of ECMWF atmospheric forecast models. ECMWF Newsletter, No. 137, ECMWF, Reading, United Kingdom, 19–22, https://www.ecmwf.int/node/17358.
Arakawa, A., and V. R. Lamb, 1977: Computational design of the basic dynamical processes of the UCLA general circulation model. Methods in Computational Physics, J. Chang, Ed., Vol. 17, Academic Press, 173–265.
Baldauf, M., 2008: Stability analysis for linear discretisations of the advection equation with Runge–Kutta time integration. J. Comput. Phys., 227, 6638–6659, https://doi.org/10.1016/j.jcp.2008.03.025.
Becker, E., and U. Burkhardt, 2007: Nonlinear horizontal diffusion for GCMs. Mon. Wea. Rev., 135, 1439–1454, https://doi.org/10.1175/MWR3348.1.
Brandt, T., 2006: A priori tests on numerical errors in large eddy simulation using finite differences and explicit filtering. Int. J. Numer. Methods Fluids, 51, 635–657, https://doi.org/10.1002/fld.1144.
Brown, A. R., S. H. Derbyshire, and P. J. Mason, 1994: Large-eddy simulation of stable atmospheric boundary layers with a revised stochastic subgrid model. Quart. J. Roy. Meteor. Soc., 120, 1485–1512, https://doi.org/10.1002/qj.49712052004.
Cooper, G., and J. Verner, 1972: Some explicit Runge–Kutta methods of high order. SIAM J. Numer. Anal., 9, 389–405, https://doi.org/10.1137/0709037.
Corbetta, G., E. Orlandi, T. Heus, R. Neggers, and S. Crewell, 2015: Overlap statistics of shallow boundary layer clouds: Comparing ground-based observations with large-eddy simulations. Geophys. Res. Lett., 42, 8185–8191, https://doi.org/10.1002/2015GL065140.
Durran, D. R., 2010: Numerical Methods for Fluid Dynamics: With Applications to Geophysics. 2nd ed. Springer, 532 pp.
Erlebacher, G., M. Y. Hussaini, C. G. Speziale, and T. A. Zang, 1992: Toward the large-eddy simulation of compressible turbulent flows. J. Fluid Mech., 238, 155–185, https://doi.org/10.1017/S0022112092001678.
Favre, A., 1983: Turbulence: Space-time statistical properties and behavior in supersonic flows. Phys. Fluids, 26, 2851–2863, https://doi.org/10.1063/1.864049.
Ghosal, S., 1996: An analysis of numerical errors in large-eddy simulations of turbulence. J. Comput. Phys., 125, 187–206, https://doi.org/10.1006/jcph.1996.0088.
Giraldo, F. X., and M. Restelli, 2008: A study of spectral element and discontinuous Galerkin methods for the Navier–Stokes equations in nonhydrostatic mesoscale atmospheric modeling: Equation sets and test cases. J. Comput. Phys., 227 3849–3877, https://doi.org/10.1016/j.jcp.2007.12.009.
Heinze, R., and Coauthors, 2017: Large-eddy simulations over Germany using ICON: A comprehensive evaluation. Quart. J. Roy. Meteor. Soc., 143, 69–100, https://doi.org/10.1002/qj.2947.
Honnert, R., and Coauthors, 2020: The atmospheric boundary layer and the “gray zone” of turbulence: A critical review. J. Geophys. Res. Atmos., 125, e2019JD030317, https://doi.org/10.1029/2019JD030317.
Kelly, J. F., and F. X. Giraldo, 2012: Continuous and discontinuous Galerkin methods for a scalable three-dimensional nonhydrostatic atmospheric model: Limited-area mode. J. Comput. Phys., 231, 7988–8008, https://doi.org/10.1016/j.jcp.2012.04.042.
Lawson, J. D., 1967: An order six Runge–Kutta process with extended region of stability. SIAM J. Numer. Anal., 4, 620–625, https://doi.org/10.1137/0704056.
Lilly, D. K., 1962: On the numerical simulation of buoyant convection. Tellus, 14, 148–172, https://doi.org/10.3402/tellusa.v14i2.9537.
Marras, S., and Coauthors, 2016: A review of element-based Galerkin methods for numerical weather prediction: Finite elements, spectral elements, and discontinuous Galerkin. Arch. Comput. Methods Eng., 23, 673–722, https://doi.org/10.1007/s11831-015-9152-1.
Mellor, G. L., and T. Yamada, 1974: A hierarchy of turbulence closure models for planetary boundary layers. J. Atmos. Sci., 31, 1791–1806, https://doi.org/10.1175/1520-0469(1974)031<1791:AHOTCM>2.0.CO;2.
Miyamoto, Y., Y. Kajikawa, R. Yoshida, T. Yamaura, H. Yashiro, and H. Tomita, 2013: Deep moist atmospheric convection in a subkilometer global simulation. Geophys. Res. Lett., 40, 4922–4926, https://doi.org/10.1002/grl.50944.
Nishizawa, S., H. Yashiro, Y. Sato, Y. Miyamoto, and H. Tomita, 2015: Influence of grid aspect ratio on planetary boundary layer turbulence in large-eddy simulations. Geosci. Model Dev., 8, 3393–3419, https://doi.org/10.5194/gmd-8-3393-2015.
Putman, W. M., and M. Suarez, 2011: Cloud-system resolving simulations with the NASA Goddard Earth Observing System global atmospheric model (GEOS-5). Geophys. Res. Lett., 38, L16809, https://doi.org/10.1029/2011GL048438.
Rogallo, R. S., and P. Moin, 1984: Numerical simulation of turbulent flows. Annu. Rev. Fluid Mech., 16, 99–137, https://doi.org/10.1146/annurev.fl.16.010184.000531.
Sato, Y., S. Nishizawa, H. Yashiro, Y. Miyamoto, Y. Kajikawa, and H. Tomita, 2015: Impacts of cloud microphysics on trade wind cumulus: Which cloud microphysics processes contribute to the diversity in a large eddy simulation? Prog. Earth Planet. Sci., 2, 23, https://doi.org/10.1186/s40645-015-0053-6.
Satoh, M., and Coauthors, 2014: The non-hydrostatic icosahedral atmospheric model: Description and development. Prog. Earth Planet. Sci., 1, 18, https://doi.org/10.1186/s40645-014-0018-1.
Satoh, M., B. Stevens, F. Judt, M. Khairoutdinov, S.-J. Lin, W. M. Putman, and P. Düben, 2019: Global cloud-resolving models. Curr. Climate Change Rep., 5, 172–184, https://doi.org/10.1007/s40641-019-00131-0.
Savic-Jovcic, V., and B. Stevens, 2008: The structure and mesoscale organization of precipitating stratocumulus. J. Atmos. Sci., 65, 1587–1605, https://doi.org/10.1175/2007JAS2456.1.
Skamarock, W. C., 2004: Evaluating mesoscale NWP models using kinetic energy spectra. Mon. Wea. Rev., 132, 3019–3032, https://doi.org/10.1175/MWR2830.1.
Skamarock, W. C., and Coauthors, 2008: A description of the Advanced Research WRF version 3. NCAR Tech. Note NCAR/TN-475+STR, 113 pp., https://doi.org/10.5065/D68S4MVH.
Skamarock, W. C., J. B. Klemp, M. G. Duda, L. D. Fowler, S.-H. Park, and T. D. Ringler, 2012: A multiscale nonhydrostatic atmospheric model using Centroidal Voronoi Tesselations and C-Grid staggering. Mon. Wea. Rev., 140, 3090–3105, https://doi.org/10.1175/MWR-D-11-00215.1.
Smagorinsky, J., 1963: General circulation experiments with the primitive equations: I. The basic experiment. Mon. Wea. Rev., 91, 99–164, https://doi.org/10.1175/1520-0493(1963)091<0099:GCEWTP>2.3.CO;2.
Sommeria, G., 1976: Three-dimensional simulation of turbulent processes in an undisturbed trade wind boundary layer. J. Atmos. Sci., 33, 216–241, https://doi.org/10.1175/1520-0469(1976)033<0216:TDSOTP>2.0.CO;2.
Stevens, B., and Coauthors, 2019: DYAMOND: The DYnamics of the Atmospheric general circulation Modeled On Non-hydrostatic Domains. Prog. Earth Planet. Sci., 6, 61, https://doi.org/10.1186/s40645-019-0304-z.
Stevens, B., and Coauthors, 2020: The added value of large-eddy and storm-resolving models for simulating clouds and precipitation. J. Meteor. Soc. Japan, 98, 395–435, https://doi.org/10.2151/jmsj.2020-021.
Tomita, H., and M. Satoh, 2004: A new dynamical framework of nonhydrostatic global model using the icosahedral grid. Fluid Dyn. Res., 34, 357–400, https://doi.org/10.1016/j.fluiddyn.2004.03.003.
Vreman, B., B. Geurts, and H. Kuerten, 1994: Discretization error dominance over subgrid terms in large eddy simulation of compressible shear layers in 2d. Commun. Numer. Methods Eng., 10, 785–790, https://doi.org/10.1002/cnm.1640101004.
Wan, H., and Coauthors, 2020: Improving time step convergence in an atmosphere model with simplified physics: The impacts of closure assumption and process coupling. J. Adv. Model. Earth Syst., 12, e2019MS001982, https://doi.org/10.1029/2019MS001982.
Wicker, L. J., and W. C. Skamarock, 2002: Time-splitting methods for elastic models using forward time schemes. Mon. Wea. Rev., 130, 2088–2097, https://doi.org/10.1175/1520-0493(2002)130<2088:TSMFEM>2.0.CO;2.
Wyngaard, J. C., 2004: Toward numerical modeling in the “terra incognita.” J. Atmos. Sci., 61, 1816–1826, https://doi.org/10.1175/1520-0469(2004)061<1816:TNMITT>2.0.CO;2.
Zängl, G., D. Reinert, P. Rípodas, and M. Baldauf, 2015: The ICON (ICOsahedral Non-hydrostatic) modelling framework of DWD and MPI-M: Description of the non-hydrostatic dynamical core. Quart. J. Roy. Meteor. Soc., 141, 563–579, https://doi.org/10.1002/qj.2378.
A series of numerical experiments were conducted in the Oakbridge-CX system at the University of Tokyo. Each experiment used 576 MPI processes, and the number of OpenMP threads was set to six per MPI process.