Spatially Local Surrogate Modeling of Subgrid-Scale Effects in Idealized Atmospheric Flows: A Deep Learned Approach Using High-Resolution Simulation Data

a. General procedure

b. Fluid flow problem

Equation (3) can be represented in vector form as ∂_tq + ∂_tf + ∂_tg = s with the state vector $\mathbf{q}={[\rho \rho u\rho w\rho \theta ]}^T$. We only model the perturbed scalars, ρ′ and [(ρθ)′], removing the dominant underlying hydrostatic balance described by Eq. (4). Although inviscid, the Euler equations form the basis of atmospheric flow simulations, and inherent numerical dissipation maintains stability. Moreover, the two-dimensional setting is an apt choice for idealized simulation as the majority of idealized test cases in the literature are two-dimensional.

We show the time evolution of potential temperature from initial condition to a turbulent state in Fig. 2. The flow is perturbed from a neutrally stratified dry atmosphere. The domain is of size (x, z) ∈ [0, 20] km × [0, 10] km. Slip, solid wall boundary conditions are prescribed for the top and bottom walls. The left and right walls are prescribed with periodic boundary conditions. We choose the grid resolution, n_x × n_z, of the coarse simulation to be 200 × 100 (grid spacing of 100 m in all directions). The fine-resolution domain has a resolution of 1000 × 500, resulting in a grid mapping ratio of 5× between the coarse and fine domains. This grid mapping leads to a fine-grid spacing of 20 m in the fine-resolution simulation. The time steps are adjusted so that the Courant–Friedrichs–Lewy number is 0.8, corresponding to a time step of 0.23 and 0.046 s for the coarse- and fine-resolution simulations, respectively. We simulate the flows until 2000 s. After the initial laminar regime dominated with antidissipative effects, the flow evolves to a highly turbulent regime from about 500 s onward, as shown in Fig. 2. The temporal regimes consisting of both antidissipative and dissipative effects make the colliding thermal test case a useful model problem to test the capability of the current framework to capture these effects. Further details of the test case, solver, and numerical schemes can be found in Norman (2021).

View Full Size

Time evolution of the colliding thermal problem from initial condition to a turbulent state portrayed using potential temperature perturbations, θ′.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide

To generate the training data for Δq, we start with the fine state q_fine and the coarse state q_coarse, which is the coarse-grained high-resolution state ${\overline{q}}_{\text{fine}}$ using a stencil of 5 × 5 corresponding to the grid mapping ratio of 5×. The coarse graining for the boundary cells is computed using ghost cell values based on the respective boundary conditions in the horizontal and vertical walls. We then advance each of these two states, q_coarse and q_fine, by a single coarse-resolution time step (five time steps for the fine simulation in the current case) and then calculate the state difference, Δq, between them. Finally, q_coarse is replaced by ${\overline{q}}_{\text{fine}}$ at the end of the coarse time step, and the procedure is repeated.

c. Network setup

We deploy a supervised deep learning technique to model the relation between the inputs, a local stencil of spatially coarse-resolution states q_coarse, and the outputs, state difference in the center cell of the stencil with the spatially fine-resolution states, Δq. Note that the mapping is only in space and not in time, and thus, the coarsening is done just in space. We model this mapping between q_coarse and Δq using deep NNs of the form $\mathcal{N}({\mathbf{q}}_{\text{coarse}},\mathbf{w})$ with w representing the parameters (“weights”) of the NN. The learning process involves optimizing the parameters w of the NN by minimizing the loss computed by the function $L[\mathrm{\Delta }\mathbf{q},\mathcal{N}({\mathbf{q}}_{\text{coarse}},\mathbf{w})]$. In the current analysis, we use a squared L₂ norm as the loss function:

$L[\mathrm{\Delta }\mathbf{q},\mathcal{N}({\mathbf{q}}_{\text{coarse}},\mathbf{w})]={\Vert \mathrm{\Delta }\mathbf{q}-\mathcal{N}({\mathbf{q}}_{\text{coarse}},\mathbf{w})\Vert }_2^2.$(5)

A stencil-based approach is used to build the architecture of $\mathcal{N}({\mathbf{q}}_{\text{coarse}},\mathbf{w})$. We use the stencil around each grid cell in the coarse simulation to model the Δq at the respective locations as shown in Fig. 3. The framework of $\mathcal{N}({\mathbf{q}}_{\text{coarse}},\mathbf{w})$ is composed of 36 inputs, the flattened vector of 4 state variables of the 3 × 3 grid stencil around a given coarse cell. The outputs (4 values) are the Δq at the given grid cell (the center of the stencil of inputs). The hidden layer(s) of the NN comprise neurons, which enable the nonlinear activation function to capture the relation between the inputs and outputs (Goodfellow et al. 2016). We compare three different types of NN architectures for the hidden layer(s) with a leaky ReLU activation function having a slope of 0.1 (Glorot et al. 2011; Maas et al. 2013):

1. A single layer of 45 neurons (single-layer model)–1849 learnable parameters

2. Ten layers with 45 neurons/layer in a ResNet-based configuration (He et al. 2016) for multilayer perceptrons, which we call the ResNet model–23 499 learnable parameters

3. Ten layers with 45 neurons/layer in a DenseNet-based configuration (Huang et al. 2017) for multilayer perceptrons, which we call the DenseNet model–132 808 learnable parameters

View Full Size

Illustration of the network model (the results discussed focus on the ResNet model) for modeling the state difference using stencil data.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide

We tune various parameters of the NN architecture, called hyperparameters, through grid search to arrive at these architectures. These hyperparameters for the current approach involve (the values chosen are provided in parenthesis):

1. Number of neurons per layer (varied between 10 and 50; 45 was chosen)

2. Number of layers (varied between 1 and 20; 10 was chosen)

3. Activation function (ReLU and LeakyReLU; LeakyReLU with 0.1 slope was chosen to enable the negative values of the states)

4. Loss function at the output layer (L₁, a combination of L₁ for large errors and squared L₂ for small errors, and squared L₂; squared L₂ or mean-squared-error loss was chosen)

5. Values for regularization at the input, hidden, and output layers (L₁ and L₂ regularization with regularization values ranging between 1 × 10⁻⁹ and 1 × 10⁻¹; although both L₁ and L₂ regularization with a value of 1 × 10⁻⁶ had the best results within the range, no regularization was eventually used as they did not affect the model accuracy and numerical stability drastically)

6. Drop-out regularization (randomly zeroing out some of the elements of the tensors with a given probability Srivastava et al. 2014) at the input and hidden layers (varied between 0 and 0.1; only the input layer was chosen to have dropout with a value of 0.01)

7. Optimization routine (stochastic gradient descent, Adam, and NAdam; NAdam was chosen due to higher convergence rate)

8. Batch size of training samples (varied between 1 and 2048; 1024 was chosen).

We note that these grid searches are exploratory in nature and there are other avenues to find the most optimized NN architecture (Balaprakash et al. 2018; Egele et al. 2021; Hertel et al. 2020).

The single-layer model is used as a shallow, simple NN model to test the capability of NNs to capture the grid mapping. Drop-out regularization (Srivastava et al. 2014) is applied to the input layer to avoid overfitting and stabilize the single-layer model. Adding more linear layers with drop-out regularization to this model (layers that are equivalent to the only layer in the single-layer model) reduced the stability of the model during testing without a considerable increase in accuracy and ability to capture the nonlinear physics. This leads us to the last two models, the ResNet and DenseNet models. The last two models are deep NNs, used to increase the complexity of the network and accurately capture the highly nonlinear physics. These deep NNs take advantage of skip connections to enable sparse, complex, nonlinear relations between the input and output (He et al. 2016). The difference between the latter two is only in the manner in which the residual (or input) for each layer is computed—the first one uses addition, whereas the second uses concatenation.

We have found that the simple, small framework of using a stencil-based input and output to the network of MLPs performs similarly using convolutional neural networks (CNNs) during testing in unknown data regions. The architecture is intrinsically similar to convolution operations. More importantly, the stencil-based approach mimics a higher-order finite difference scheme. The current formulation does not assume a dissipation- or advection-based mapping for the relationship between q_coarse and Δq. Our approach is to let the NN capture the complex mathematical relations between the states of the center cell and neighboring cells with the state difference at the center cell in the coarse domain.

It is important to note that the actual domain of constraint for the fine-scale model, the domain of inputs used to predict the outputs, is wider than the 3 × 3 coarse cells over the temporal domain of a coarse-grid time step. Therefore, full constraint of the state difference is likely not possible. Still, we wished to maintain a simpler NN model with minimal inputs, and results will show that the 3 × 3 stencil is effective in providing a constraining set of inputs.

d. Training and validation of model

We have learned that sampling the data for training is a crucial step for building an accurate and stable NN model, which we elaborate on here. The training dataset comprises 1-M samples collected between the temporal regime of [0, 900] s. We did a preliminary grid search on the number of training samples required, spanning from a few 100-k to 10-M samples, and found 1-M samples to be enough for training accurate models. Note that this sampling set is obtained using less than 2% of the spatial data at each time step of the simulation. For comparison, the number of training data points generated for a full flow field model (like CNN models) would be over 313 M over the same temporal regime. The small stencil-based framework allows us to use significantly fewer samples for training the models.

To alleviate this bias issue, we use a 50:50 split to curate samples with and without high TV weighting. To sample n = 10⁶/N_TS data points at an instant in time, where N_TS is the number of time steps, we apply a strict sampling criterion using TV of all the states of a data point. First, we perform a min–max normalization of V(q) for each state computed over the global spatial domain for that instant in time. Next, a threshold is selected to identify n/2 random samples with high TV weighting for all the states. Selecting this threshold poses further challenges as the flow evolves through both laminar and turbulent regimes, leading to a broad range of values for the states over time. Thus, the threshold for identifying the 50:50 split is manually adjusted for various flow regimes. Finally, the rest of the n/2 samples at an instant in time are randomly sampled from the remaining data points in the domain.

We train the model in a Python environment using PyTorch and then deploy the model in the C++ solver (Norman 2021) with graphics processing unit (GPU) acceleration for production usage. The “NAdam” optimizer is used for optimizing the weights of the NNs (Kingma and Ba 2014; Dozat 2016; Ma and Yarats 2019). For training the 1-M data points (with a 70:30 split between training and validation loops and random shuffling of training samples), we use minibatches of 1024 samples. Even with the use of minibatches, the model incurs high variance in accuracy over the training epochs. We have learned that using a higher learning rate at initial training epochs till saturation of the model accuracy and then lowering the value help reduce this high variance in accuracy. Thus, this process of reducing the learning rate represents a two-step reduction, and we use high and low learning rate values of 1 × 10⁻¹ and 5 × 10⁻², respectively. Furthermore, for the deep NNs, we also rely on an averaging technique to ensemble the model weights across various training epochs to alleviate the high variance in the training accuracy as shown in Fig. 4. After a regime of stable accuracy is reached over epochs (3000 epochs is set for our current problem), we collect ensembles of model weights from models with errors below a particular threshold (a mean-square-error value of 2.5 × 10⁻⁴ is used). The weights of the final model are an ensemble of these collected weights. The predictions using the NN are made only using this single final model, whose weights are computed from the ensembling procedure. The procedure helps average the gradient basin in the optimization manifold, aiding in forming a stable solution. This procedure has similarities with the stochastic weighted averaging (SWA) method introduced by Izmailov et al. (2018).

View Full Size

(top) Illustration of collecting ensembles to form ensemble model weights [similar to the stochastic weighted averaging technique by Izmailov et al. (2018)]. The green line represents the training regime over which the ensemble of the model weights is computed. The red shaded region denotes the training instances when the model acquires error values higher than the cutoff error and hence the weights of which are not considered for the ensembling procedure. (bottom) A sample training curve for the current modeling framework.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide

a. Offline testing

For the offline testing, the results from the NN model are directly compared with the expected solution and evaluated independently without coupling the NN model with the numerical solver. We first test the model with data randomly sampled in space and time outside the training regime, as shown in Fig. 5. The comparison between the true and predicted Δq for each flow state is shown in Figs. 5a–d, along with the coefficient of determination (regression score), R². The ResNet model predicts the state differences with high accuracy and low spread, with R² > 0.95. The normalized errors have a mean and standard deviation of $\mathcal{O}({10}^{-3})$ and $\mathcal{O}({10}^{-2})$, respectively.

View Full Size

Comparison between the true state differences and those predicted by the ResNet-based NN model for random data points in space and time. The coefficient of determination, R² value, for each state difference is also provided.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide

Next, we observe the behavior of the model by using data from a full snapshot in a turbulent regime outside the temporal regime of the training data (after 900 s), as shown in Fig. 6 for Δ(ρu). The ResNet model predicts the state difference accurately with an overall Euclidean error norm of $L_2={\Vert \mathrm{\Delta }\mathbf{q}-\mathrm{\Delta }{\mathbf{q}}_{\text{NN}}\Vert }_2/{\Vert \mathrm{\Delta }\mathbf{q}\Vert }_2=0.195$. We also plot the L₁ error, ϵ = |Δq − Δq_NN|/max(|Δq|), to study the results locally in space, as shown in Fig. 6 (bottom). The main large-scale structures are accurately captured, whereas the smaller-scale features have lower accuracy. Nonetheless, the maximum error is 10%, which is isolated to a small region in the middle-left side of the domain. The DenseNet model performed accurately for both the random samples [normalized errors have a mean and standard deviation of $\mathcal{O}({10}^{-3})$ and $\mathcal{O}({10}^{-2})$, respectively] and full flow field (L₂ = 0.55) test datasets. Surprisingly, the single-layer model did give accurate results [normalized errors have a mean and standard deviation of $\mathcal{O}({10}^{-2})$ and $\mathcal{O}({10}^{-2})$, respectively, and L₂ = 1.32]. However, neither the DenseNet nor the single-layer models performed as well as the ResNet model.

View Full Size

Full flow field prediction by the NN model. Corrections of horizontal velocity are shown along with the L₁ error norm, ϵ = |Δq − Δq_NN|/max(|Δq|), and L₂ norm of the full flow field, $L_2={\Vert \mathrm{\Delta }\mathbf{q}-\mathrm{\Delta }{\mathbf{q}}_{\text{NN}}\Vert }_2/{\Vert \mathrm{\Delta }\mathbf{q}\Vert }_2$.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide

b. Online testing

The online testing is done by coupling the NN model correction with the coarse flow simulation. A sample depiction of the NN-coupled solver is shown in Fig. 7. The fine simulation is used as correction until the testing time, t₀, after which the NN model is invoked. Therefore, at time t₀, the model state is essentially perfect with respect to the fine-grid model. The solution at t₀ can also be viewed as the coarse-grained fine-resolution simulation result. We use the PyTorch C++ API¹ to couple the NN model with the C++ solver, performing the in-the-loop ML integration with the scientific solver. A GPU kernel is used to update the q_coarse with the machine learned Δq_NN after every time step, following Eq. (2). Even without special batching consideration of online inference and refactoring the code for GPU performance, we attain speed-ups up to 8× when using the NN-coupled solver compared to a fine-resolution simulation. Since the current discussion is concentrated more on the scientific aspects of the model, we reserve the full computational aspects of the framework for future investigation. In the following results, we show the results of testing the NN-coupled solver in a turbulent regime as well as in a laminar regime.

View Full Size

Sample depiction of online testing using the NN model.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide

We invoke the NN model at time t₀ = 1000 s, which is outside the training regime and well within the turbulent regime. Recall that the training data were sampled from t ∈ [0, 900] s. The results we show here are those after the simulation has run for t₀ + 5 s, which is approximately 25 coarse-grid time steps. The results for ρu of the coarse-grained fine-resolution simulation (${\overline{\rho u}}_{\text{fine}}$), coarse-resolution simulation without correction (ρu_{coarse-No  correction}), and coarse-resolution simulation with correction using the NN model (ρu_coarse-NN) are shown in Fig. 8. The overall accuracy of the NN model, quantified by the L₂ norm, is similar to that of the ρu_{coarse-No  correction}. We focus on the eddy at the bottom-right corner of the domain to exemplify the prominence of the diffusive effects by a coarse-grid resolution in the turbulent regime. Without correction after t₀ (ρu_{coarse-No correction}, Fig. 8 middle), diffusive effects start to dominate the coarse simulation, and the subgrid-scale effects are not captured. The sharp boundaries of the coherent structure are smoothed in ρu_{coarse-No  correction} by the coarse grid as compared to the fine grid in ${\overline{\rho u}}_{\text{fine}}$. The vortex core is predicted with reasonable accuracy in ρu_{coarse-No  correction}, but the fine-scale boundaries (or gradients) in the boundary of the vortex core and the shear layer regions are not captured.

View Full Size

Full flow state prediction using NN-based emulation in turbulent regime outside of training regime. Snapshots of horizontal velocity (ρu) after 5 s from the initial state of the testing regime are shown for (top) coarse-grained fine-resolution simulation (${\overline{\rho u}}_{\text{fine}}$), (middle) coarse simulation without correction (ρu_{coarse-No  correction}), and (bottom) coarse simulation with correction using NN model (ρu_coarse-NN). The L₂ difference norm is also given for ρu_{coarse-No  correction} and ρu_coarse-NN. The zoom-in view of the eddy at the bottom-right corner of the domain is shown adjacent to the respective contour plots. The L₁ difference with respect to ρu_fine, $ϵ=\left|{\overline{q}}_{\text{fine}}-{\mathit{q}}_{\text{coarse-No}\hspace{0.17em}\text{correction}\hspace{0.17em}\text{or}\hspace{0.17em}\text{NN}}\right|/\max (\left|{\overline{q}}_{\text{fine}}\right|)$, is also shown in the right columns. The respective contour plots of the flow fields of ρw, ρ, and θ′ are provided in the online supplemental material document.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide

The NN model, ρu_coarse-NN, accurately predicts the flow evolution with sharp boundaries. The sharp boundaries in the shear-layer regions above and below the vortex core are captured better in ρu_coarse-NN compared to ρu_{coarse-No  correction}. Moreover, the NN model maintains a maximum L₁ difference of 6% and better than that of ρu_{coarse-No  correction} (7%), as shown in Fig. 8 bottom right. Note that the chaotic divergence of the modeled flow and its difference from the ideal states are inseparably combined in the results of the L₁ difference norm. The main coherent structures are accurately captured in ρu_coarse-NN, and the larger L₁ errors from the small-scale fluctuations are isolated. Note that these regions of high local errors are located in high-gradient regions.

To provide a clear quantitative measure of the capability of the NN model to capture the sharp boundaries or high-gradient regions, we compute the gradients of ρu in both x and z directions, as shown in Figs. 9 and 10. The L₂ difference norm of the gradients with respect to that of ${\overline{\rho u}}_{\text{fine}}$ is also computed to quantify the overall difference in the gradients. Both the gradients ∇_x(ρu_coarse-NN) and ∇_z(ρu_coarse-NN) have less L₂ difference compared to that of ρu_{coarse-No  correction}. The comparatively low local errors of ρu_coarse-NN, depicted by the contours of L₁ error, also portray the ability of the NN model to capture the high gradients in the flow field. Since the L₂ norm also suffers the effects of the chaotic divergence of the flow and spatial repositioning of the fine-scale features, we further use the probability density distribution of the gradients to compare the three simulations.

View Full Size

(top) Comparison of gradients of ρu in the x direction for (left) coarse-grained fine-resolution simulation (${\overline{\rho u}}_{\text{fine}}$), (middle) coarse simulation without correction (ρu_{coarse-No  correction}), and (right) coarse simulation with correction using NN model (ρu_coarse-NN). The L₂ difference norm is also given for ρu_{coarse-No  correction} and ρu_coarse-NN. (Bottom) The L₁ difference with respect to ρu_fine, $ϵ=\left|{\nabla }_x({\overline{q}}_{\text{fine}})-{\nabla }_x({\mathbf{q}}_{\text{coarse-No}\hspace{0.17em}\text{correctionor}\hspace{0.17em}\text{NN}})\right|/\max [\left|{\nabla }_x({\overline{q}}_{\text{fine}})\right|]$. The respective contour plots of the flow fields of ρw, ρ, and θ′ are provided in the supplemental material document.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

(top) Comparison of gradients of ρu in the z direction for (left) coarse-grained fine-resolution simulation (${\overline{\rho u}}_{\text{fine}}$), (middle) coarse simulation without correction (ρu_{coarse-No  correction}), and (right) coarse simulation with correction using NN model (ρu_coarse-NN). The L₂ difference norm is also given for ρu_{coarse-No correction} and ρu_coarse-NN. (bottom) The L₁ difference with respect to ρu_fine, $ϵ=\left|{\nabla }_z({\overline{q}}_{\text{fine}})-{\nabla }_z({\mathbf{q}}_{\text{coarse-No}\hspace{0.17em}\text{correctionor}\hspace{0.17em}\text{NN}})\right|/\max [\left|{\nabla }_z({\overline{q}}_{\text{fine}})\right|]$. The respective contour plots of the flow fields of ρw, ρ, and θ′ are provided in the supplemental material document.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide

The probability density distributions of the gradients ∇_x(ρu) and ∇_z(ρu) for the three simulations are shown in Fig. 11. The NN model enables the distributions of both ∇_x(ρu) and ∇_z(ρu) to better match that of ${\overline{\rho u}}_{\text{fine}}$, particularly for high-gradient values. The distributions of the gradients of ρu_{coarse-No  correction} fall below that of ${\overline{\rho u}}_{\text{fine}}$ for large values. The Kullback–Leibler (KL) divergence (Kullback and Leibler 1951; Csiszár 1975) is used to compare the probability distributions, which for distributions q and $\widehat{q}$ is given by

$D_{\text{KL}}(q,\widehat{q})=\{\begin{array}{ll}q\log (q/\widehat{q})-q+\widehat{q}\hfill & q>0,\widehat{q}>0\hfill \\ \widehat{q}\hfill & q=0,\widehat{q}\ge 0\hfill \\ \infty \hfill & \text{otherwise}\hfill \end{array},$(8)

where the function is nonnegative and is jointly convex in q and $\widehat{q}$ (Boyd and Vandenberghe 2004). The observations regarding the distributions are quantified by the lower values of KL divergence for the ρu_coarse-NN distributions compared to ρu_{coarse-No  correction}.

View Full Size

Comparison of the probability density distributions of the gradients of (left) ∇_x(ρu) and (right) ∇_z(ρu) for coarse-grained fine-resolution simulation (${\overline{\rho u}}_{\text{fine}}$, black dash), coarse simulation without correction (ρu_{coarse-No  correction}, blue dash), and coarse simulation with correction using NN model (ρu_coarse-NN, orange dash). The Kullback–Leibler divergence, D_KL, of the ρu_{coarse-No  correction}, and ρu_coarse-NN distributions with respect to ${\overline{\rho u}}_{\text{fine}}$ are also denoted.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide

We summarize the L₂ difference norm and the KL divergence values with respect to the ${\overline{\rho u}}_{\text{fine}}$ for all the flow variables in Table 1. As seen from the L₂ values, the NN model has a higher accuracy at predicting the gradients of velocities ρu and ρw but has lower accuracy for the gradients of density ρ and potential temperature perturbations θ′. The increase in L₂ error for θ′ and ρ by the NN model could suggest that the hydrostasis might not have been captured since both these state variables are dominated by hydrostasis. The comparison of KL divergence of the gradients shows that the NN-model-based simulation outperforms the coarse simulation without correction at capturing sharp gradients for all flow variables. Next, we address the ability of the model to scale with spatial-grid resolution other than for which it was trained, which could suggest the model is generalizable to other resolutions.

Table 1.

Comparison of ∇_x(⋅) and ∇_z(⋅) for q_{coarse-No correction} (abbreviated NC) and q_coarse-NN using the L₂ difference norm and Kullback–Leibler divergence, D_KL, with respect to the coarse-grained fine-resolution simulation. The results are for online testing performed from t₀ = 1000 s, and the metrics are computed at t₀ + 5 s. The values in bold indicate comparatively lower error.

View Table

We test the generalization of the model for various grid resolutions compared to that used for training. This is considering ideas of scale invariance for the NN model (and potentially the dynamics themselves). We perform online testing of the model trained using data from the 1000 × 500 → 200 × 100 grid mapping on the following grid mappings: 1) 500 × 250 → 100 × 50, 2) 1500 × 750 → 300 × 150, and 3) 2000 × 1000 → 400 × 200. Note that the grid mapping ratio of 5× is retained for each of the mappings. The total error at an instant in a turbulent regime, at t₀ = 5 s, is shown in Fig. 12. The θ′ fields are also shown as insets for reference. The accuracy of the NN model for all the flow states scales according to the grid mapping—the error increases linearly with an increase in grid resolution.

View Full Size

Behavior of the model accuracy at online testing with simulation setups of various grid resolutions. The model was trained with stencil data from the 1000 × 500 → 200 × 100 grid mapping. All other cases have the same grid-mapping ratio of 5× between the respective coarse and fine grids. The total L₁ error is given by $\varphi =\left[1/\left({\displaystyle {\sum }_{i,j}^{n_x,n_z}\left|{\overline{q}}_{{\text{fine}}_{i,j}}\right|}\right)\right]{\displaystyle {\sum }_{i,j}^{n_x,n_z}\left|{\mathbf{q}}_{{\text{NN}}_{i,j}}-{\overline{q}}_{{\text{fine}}_{i,j}}\right|}$, where ${\overline{q}}_{\text{fine}}$ is the interpolated fine-resolution state (coarse-grained fine-resolution simulation) corresponding to each of the tested coarse grid. Insets are the coarse-grid potential temperature perturbation fields for each grid mapping.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide

These results suggest various advantages of the stencil-based architecture used in the current modeling framework. The NN model trained at a lower-grid resolution can be used at a higher resolution, though with slightly more error, hence saving significant computational costs during training. Moreover, such a scaling would be difficult to achieve if the surrogate model has global spatial and/or temporal dependence, which are not considered in the current local, stencil-based framework. While we attribute the reasoning for the scaling to the stencil-based approach, resembling a finite difference scheme, we have yet to fully understand the mechanism behind this character of the model, and this investigation will be performed in future work.

With spatial accuracy comes the next challenge of temporal stability of the NN-coupled solver. The high spatial accuracy attained by the NN does not remain numerically stable indefinitely. As the flow evolves, the simulation becomes unstable, which laminar results will show is most likely attributable to new and growing extrema produced by the NN model (see Fig. 17). The root-mean-square (rms) value of wind speeds and total kinetic energy over time are shown in Fig. 13. The rms value (spatial) of wind speeds and the total kinetic energy are defined by

${\mathbf{u}}_{\text{rms}}={\left[\frac{1}{n_x{n}_z}{\displaystyle \sum _{i,j}^{n_x,n_z}{\mathbf{u}}_{i,j}^2}\right]}^{1/2}\hspace{0.17em}\text{and}\hspace{0.17em}\text{Total}\hspace{0.17em}\text{KE}=\frac{1}{2}\left[{\displaystyle \sum _{i,j}^{n_x,n_z}{(\rho u)}_{i,j}^2}+{(\rho w)}_{i,j}^2\right],$(9)

respectively, where $\mathbf{u}={[\rho u\rho w]}^{\mathrm{T}}$. We also plot the L₁ difference of the rms velocity and total KE with respect to the coarse-grained fine simulation on the right column of Fig. 13, computed as ϵ = |(u_{rms,coarse-NN} − u_rms,fine)/u_rms,fine|. Overestimates of variability, as seen previously in L₁ difference plots of Fig. 8, accumulate over time until the solver becomes unstable at t₀ + 20 s, which is after ∼90 coarse-grid time steps. However, in the stable regime, the (ρu)_rms of the coarse simulation with NN correction better maintains the average rms of the field over time compared to the coarse simulation without correction (see blue lines in Fig. 13, top right). The accuracy of the (ρw)_rms of the coarse simulation with NN correction is comparable to that of the coarse simulation without correction. The total KE (Fig. 13, bottom right) shows that the NN-based correction indeed outperforms the coarse simulation without correction during most of the stable temporal regimes.

View Full Size

Comparison of (top) the rms values of the flow velocities and (bottom) the total kinetic energy over time. The various lines represent the following: – the coarse-grained fine-resolution simulation, ⋅⋅⋅ the coarse simulation without correction from t₀ = 1000 s, and – – the coarse simulation with NN correction. (left) The raw data and (right) the L₁ difference with respect to the coarse-grained fine-resolution simulation.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide

The results suggest that the NN-based correction is highly accurate over a finite time interval of ∼90 coarse-grid time steps, whereas further research needs to be done to make the solver stable from the accumulation of the errors at high-gradient regions. We note that the DenseNet model is able to perform similarly to that of the ResNet model, becoming unstable after t₀ + 12 s but with higher levels of error before it becomes unstable. However, the single-layer model is highly unstable, as it becomes unstable after only t₀ + 3 s. This inability of the single-layer model could suggest the utility of a complex formulation of the NN model with multiple layers and skip connections.

Various regularization techniques can be used to address these overestimates although we did not see many changes with such strategies. Utilizing the skip connections in ResNets was one of the ways in which the overestimates were controlled, but not fully alleviated. Since the overestimates do not satisfy the physics constraints of the fluid flow system, physics-informed techniques can be used to constrain the variabilities. We provide insights from our ongoing work on improving the stability of the NN-coupled solver in the conclusions section.

Testing in the laminar regime

Testing the model in a turbulent regime shows its capability to capture predominantly dissipative effects. Next, we demonstrate the capability of the NN model to effectively capture predominantly antidissipative effects by testing it in a transient regime dominated by laminar flow where mixing is not prevalent. For this testing in a laminar regime, we invoke the NN model from t₀ = 300 s. The results after the simulation has run for t₀ + 5 s are shown in Fig. 14 (similar to the plots in turbulent regime, Fig. 8). The NN model is able to accurately predict the corrections in laminar regimes as well, with a difference (L₂ = 0.009) lesser than the difference acquired by the coarse simulation without correction (L₂ = 0.013).

View Full Size

Full flow state prediction using NN-based emulation in laminar regime outside of training regime. Snapshots of potential temperature perturbation (ρu) after 5 s from the initial state of the testing regime are shown for (top) coarse-grained fine-resolution simulation (${\overline{\rho u}}_{\text{fine}}$), (middle) coarse simulation without correction (ρu_{coarse-No correction}), and (bottom) coarse simulation with correction using NN model (ρu_coarse-NN). The L₂ difference norm is also given for ρu_{coarse-No correction} and ρu_coarse-NN. The L₁ difference with respect to ρu_fine, $ϵ=\left|{\overline{q}}_{\text{fine}}-{\mathit{q}}_{\text{coarse-No}\hspace{0.17em}\text{correction}\hspace{0.17em}\text{or}\hspace{0.17em}\text{NN}}\right|/\max (\left|{\overline{q}}_{\text{fine}}\right|)$, is also shown in the right column. The respective contour plots of the flow fields of ρw, ρ, and θ′ are provided in the supplemental material document.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide

We also plot the gradient of ρu in both x and z directions as shown in Figs. 15 and 16. The NN correction outperforms the coarse simulation without corrections for both ∇_xρu and ∇_zρu. Moreover, Table 2 shows the NN correction to outperform the coarse simulation without correction at capturing the high gradients for all the variables. These results in the laminar regime demonstrate the capability of the NN model to accurately capture high gradients in a flow regime not dominated by mixing and diffusion. Moreover, note that the NN model is accurate for both ρu and θ′ unlike the results in the turbulent regime. This observation suggests that we need to be more careful when choosing the variables being modeled and include the effects of hydrostasis along with the perturbed variables. Moreover, the observation could also suggest the NN model considers mixing in the vertical direction similar to that in the horizontal direction. Thus, a more anisotropic framework may suit the current flow setup.

View Full Size

(top) Comparison of gradients of ρu in the x direction at the laminar regime for (left) coarse-grained fine-resolution simulation (${\overline{\rho u}}_{\text{fine}}$), (middle) coarse simulation without correction (ρu_{coarse-No correction}), and (right) coarse simulation with correction using NN model (ρu_coarse-NN). The L₂ difference norm is also given for ρu_{coarse-No correction} and ρu_coarse-NN. (bottom) The L₁ difference with respect to ρu_fine, $ϵ=\left|{\nabla }_x({\overline{q}}_{\text{fine}})-{\nabla }_x({\mathbf{q}}_{\text{coarse-No}\hspace{0.17em}\text{correctionor}\hspace{0.17em}\text{NN}})\right|/\max [\left|{\nabla }_x({\overline{q}}_{\text{fine}})\right|]$. The respective contour plots of the flow fields of ρw, ρ, and θ′ are provided in the supplemental material document.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

(top) Comparison of gradients of ρu in the z direction at the laminar regime for (left) coarse-grained fine-resolution simulation (${\overline{\rho u}}_{\text{fine}}$), (middle) coarse simulation without correction (ρu_{coarse-No correction}), and (right) coarse simulation with correction using NN model (ρu_coarse-NN). The L₂ difference norm is also given for ρu_{coarse-No correction} and ρu_coarse-NN. (bottom) The L₁ difference with respect to ρu_fine, $ϵ=\left|{\nabla }_z({\overline{q}}_{\text{fine}})-{\nabla }_z({\mathbf{q}}_{\text{coarse-No}\hspace{0.17em}\text{correctionor}\hspace{0.17em}\text{NN}})\right|/\max [\left|{\nabla }_z({\overline{q}}_{\text{fine}})\right|]$. The respective contour plots of the flow fields of ρw, ρ, and θ′ are provided in the supplemental material document.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide

Table 2.

Comparison of ∇_x(⋅) and ∇_z(⋅) for q_{coarse-No correction} (abbreviated NC) and q_coarse-NN using the L₂ difference norm and Kullback–Leibler divergence, D_KL, with respect to the coarse-grained fine-resolution simulation. The results are for online testing performed from t₀ = 300 s, and the metrics are computed at t₀ + 5 s. The values in bold indicate comparatively lower error.

View Table

We analyze the states in a one-dimensional space, which helps reveal the performance of the model at capturing sharp gradients in the laminar regime and, hence, demonstrates the ability of the model to capture antidissipative effects. For the above online test results at time t₀ + 5 s, we analyze the flow states at a horizontal slice at z = 6 km as shown in Fig. 17. We chose the slice at z = 6 km carefully, where sharp gradients are present. The results show that the NN model (orange lines) is able to capture the most highly variable regions more accurately than the noncorrected coarse simulation (blue lines), e.g., at x = 6, 15 km in Fig. 17a. However, in some regions such as z ∈ [11, 13] km, the NN-corrected coarse simulation performs worse than the noncorrected coarse simulation. Nonetheless, even though q_{coarse-No correction} captures the sharp gradients over the short testing time period of 5 s, these will eventually be smoothed (diffused) over a finite period of time. The results of q_coarse (coarse simulation starting from t₀ = 0 s without any correction; green lines) demonstrate this inability of the coarse simulation to retain the sharp gradients over time. All the above results suggest that the NN model has the capability of capturing and steepening gradients when appropriate, particularly in regions not dominated by mixing.

View Full Size

Slices of flow states at z = 6 km from the results in Fig. 14 at time t = t₀ + 5 s. The line legends correspond to the following: dashed line (– –) is q_fine, the blue line is q_{coarse-No correction}, the orange line is q_coarse-NN, and the green line is q_coarse, the coarse simulation without any correction from the initial condition at t = 0 s.

Citation: Artificial Intelligence for the Earth Systems 3, 4; 10.1175/AIES-D-23-0043.1

• Download Figure
• Download figure as PowerPoint slide