a. SR with neural networks

Recently, deep convolutional neural networks (CNNs) have been applied to SISR and have significantly outperformed past algorithms. Initially, Dong et al. (2016) used a three-layer CNN to achieve state of the art SISR results, and the approach was quickly expanded to use significantly deeper CNNs (Kim et al. 2016). SISR-CNN architectures have rapidly developed, and complex SISR networks are now built up of many blocks of convolutional layers that include skip connections such as dense blocks (Huang et al. 2017), residual blocks (He et al. 2016), and channel attention blocks (Bastidas and Tang 2019). The CNNs’ internal spatial upsampling operators (which perform the resolution increases) have progressed from bicubic upsampling (Dong et al. 2016), to learned kernels (Long et al. 2014), to the “pixel-shuffle” approach (Shi et al. 2016). The loss functions have also evolved from simple pixelwise errors, to feature loss (based on features learned by image classification CNNs) and adversarial loss (Ledig et al. 2017), which allow the CNNs to “hallucinate” plausible subpixel-scale features (i.e., generate features that could not be directly inferred from the LR data). Current CNN SR schemes combine many of these concepts (Anwar and Barnes 2019). Wang et al. (2020) review CNN-based SISR.

b. Invertible super-resolution networks

SISR-CNNs are not typically invertible. Training them usually involves degrading HR images and tasking the CNN to reconstruct them, but applying the same degradation to the CNN output does not necessarily reproduce the input image. SISR is an ill-posed problem because there are usually multiple HR images that produce the same LR image when downsampled, and constraining SISR-CNN outputs to this manifold of possible HR images is desirable (Menon et al. 2020). Several studies have approximated this type of invertibility with a modified loss function that computes the pixel error between the downsampled output and the LR input (Menon et al. 2020; Abdal et al. 2019; Ulyanov et al. 2018; Zhang et al. 2020). Additionally, in cases in which the HR image is known, but needs to be intentionally degraded (image compression), training two CNNs simultaneously to perform upsampling and downsampling provides better performance than other SR schemes (Sun and Chen 2020; Kim et al. 2018). These approaches only approximate invertibility, however, and may not be sufficient for “super resolving” scientific datasets.

c. Downscaling in the environmental sciences

In the environmental sciences, mapping data from a low-resolution grid to a high-resolution grid is often referred to as “downscaling,” and CNN-based SR can be seen as a type of statistical downscaling (Wilby et al. 2004). To clarify terminology: “downscaling” should not to be confused with “downsampling,” which, as used here, involves reducing the resolution of data. “Downscaling” is a broad term, and there are other types of downscaling such as dynamical downscaling, in which a high-resolution simulation is used to enhance a low-resolution simulation (Xu et al. 2019; Wang et al. 2021), or model output statistics (Glahn and Lowry 1972) where gridded weather model output is mapped to point forecasts. In these scenarios, strict enforcement of invertibility is not necessarily useful, but in the context of statistical downscaling that maps low-resolution data to a high-resolution grid our method can ensure strict preservation of the physical quantities in the input field.

d. Contributions and impacts

This study introduces a method—referred to here as “downsampling enforcement” (DE)—that strictly constrains a super-resolution CNN’s output to be exactly invertible under 2D-average downsampling. This is accomplished using a transfer function that is applied after the last convolutional layer of the CNN. We demonstrate the method on scientific datasets from a weather radar, satellite imager, and climate model, all of which are cases in which strictly enforcing conservation of quantities in the low-resolution input data is potentially important, and we find that it improves the performance of the CNNs in each case. We then demonstrate this method using seven different CNN SISR architectures on five common benchmark image datasets, and show that it consistently improves performance in more typical SISR applications.

e. Motivation

There are many scientific fields where CNN-based SR could be applied to image data or, more generally, regularly gridded datasets. In these applications, guaranteed invertibility under 2D-averaging can ensure the SR scheme’s inputs and outputs are physically consistent. For example, satellite imagers often have a much larger dynamic range than handheld cameras and undergo rigorous calibration and validation to ensure that measured radiances are accurate (Markham et al. 2014); this should be considered when applying CNN-based SR (Liebel and Körner 2016; Lanaras et al. 2018; Müller et al. 2020). Other possible applications of this method include data from ranging instruments such as radars (Geiss and Hardin 2020), sonars (Feldens 2020), and lidars (Liu et al. 2020). Weather radars can be used to estimate precipitation rates for instance (Fulton et al. 1998), a physical quantity that should be conserved under spatial averaging. Super resolution can be used to enhance output from gridded numerical simulations (Baño-Medina et al. 2020), and CNN-SISR has already been demonstrated on several real-world numerical simulation problems, including: precipitation modeling (Wang et al. 2021) (who focus on dynamical downscaling), wind and solar modeling (Stengel et al. 2020), and climate modeling (Vandal et al. 2018). In climate simulations, strict conservation of physical quantities in super-resolved fields is of particular importance because climate signals can be relatively weak. Also, if SR is used during model integration, even small errors can grow rapidly over many time steps and significantly impact results. The lack of strictly enforced physical consistency in CNNs, and internal representation of physics in general, has been identified as a major hurdle that must be addressed before their impressive capabilities can be fully brought to bear on important imaging and modeling problems in the physical sciences (Reichstein et al. 2019; Tsagkatakis et al. 2019; Beucler et al. 2021; McGovern et al. 2021). In these SISR applications, and many others, strict conservation of large-scale statistical properties in the LR input fields is often just as important as the visual fidelity of the HR outputs, and our method can ensure both.

a. Neural network

For each dataset, we use the Enhanced Deep Residual Network (EDRN) architecture (Lim et al. 2017). EDRN is based on ideas from He et al. (2016) and is composed of a series of residual blocks. These blocks pass their input through several convolutional layers and then add the result to the original input. The intuition behind this is that residual skip connections provide a more direct path between the input and output of the network and allow the neural network to learn which residual blocks and which features to use as it trains, combating the vanishing gradient problem (the tendency for gradients to approach zero in very deep neural networks during training, which prevents them from learning). “Enhanced” refers to removing several aspects of the residual blocks that were used in He et al. (2016) that are not beneficial for SR (batch normalization for instance). We note that, in the examples of SR applied in the physical sciences given in section 1e, EDRN is the most commonly used model framework, with Stengel et al. (2020), Lanaras et al. (2018), and Liu et al. (2020) all using EDRN-like architectures. Our implementation has 16 residual blocks with 128-filter convolutional layers. The CNN operates at low resolution and then upsamples using a pixel shuffle (Shi et al. 2016) as one of the final operations in the network. The implementation used here has 5.2 × 10⁶ trainable parameters.

b. Training

This study uses 2D-average downsampling for image and data degradation. Bicubic downsampling with antialiasing is more common when working with images in the SISR literature—specifically the MATLAB scheme—but 2D-averaging was assumed in deriving (2) because this is the more common physical relation between high- and low-resolution data. The CNNs here perform 4× SISR, converting 48 × 48 pixel inputs to 192 × 192 outputs. Inputs are standardized to a [−1, 1] scale, and a tanh activation is applied to the output. Outputs are then scaled back to the appropriate (dimensional) range. In the DE cases the tanh activation is applied before applying (2). Each CNN is trained for 300 epochs, with the learning rate reduced by a factor of 10 after the 200 epoch. Epochs are 1000 batches of 16 image chips selected randomly from the training set with random flips and rotations. Pixelwise mean-square error (MSE) is used as a loss function, and the Adam optimizer is used with an initial learning rate of 10⁻⁴, β₁ = 0.9, and β₂ = 0.999. A lower initial learning rate of 2 × 10⁻⁵ was used for the Storm Event Imagery (SEVIR) data because the model did not successfully train with the higher learning rate.

c. GOES-17 L1b radiance

The Geostationary Operational Environmental Satellite 17 (GOES-17) (NOAA 2019a) is a geostationary satellite currently orbiting above the equatorial Pacific at 137.2°W. Here, we apply SR to level-1b radiance data from the Advanced Baseline Imager band 2 (the red 640-nm band), which has a resolution of 0.5 km at nadir (NOAA 2019b). The images selected for this study are full-disk scans taken near 1200 LST on various days during 2019 and 2020. The images are cropped to pixels between 2452 and 19 348 in the height dimension and from 8100 to 17 700 in the width dimension to avoid extreme viewing angles and low illumination near the Earth’s limb. The exact file names for the training and test data are given in section 3 of the online supplemental material. The L1b radiances have units of watts per meter squared per steradian per micrometer, and enforcing strict conservation of the low-resolution radiances yields a slight performance improvement for the SR scheme (Fig. 3a). Example outputs are shown in Figs. 2a–c. The final RMSE on the test set was 13.72 W m⁻² sr⁻¹ μm⁻¹ for the conventional CNN and 13.71 W m⁻² sr⁻¹ with DE. The RMSE between the inputs and downsampled outputs for the conventional CNN was 0.20 W m⁻² sr⁻¹ μm⁻¹. For the DE CNN, this value is mathematically constrained to be zero though in practice it is a very small nonzero number whose magnitude depends on the floating point precision used when evaluating the CNN.

d. ERA5 cloud fraction

The fifth major global reanalysis produced by the European Centre for Medium-Range Weather Forecasts Reanalysis (ERA5) is a reconstruction of the past state of the atmosphere from 1979 to present. The reanalysis is performed by assimilating historical atmospheric observations with a numerical weather model (Hersbach et al. 2020). Here we apply SR to cloud fraction data that represent the fraction (0%–100%) of each model gridcell area covered by cloud. We use daily 0.25° × 0.25° resolution data between latitudes ±45° at 0000 UTC for the period from 1 January 1979 to 31 December 2018 for training and use data from 2019 for testing. Note that these data are on a latitude–longitude, grid whereas our DE technique assumes an equal-area grid, so here we have used only data near the equator where there is less distortion. It is possible to modify the DE technique to include latitude weightings, but we leave this for future work. Nonetheless, enforcing physical consistency between the LR and HR data in this context provides a small performance improvement for the SR scheme (Fig. 3b). Examples are shown in Figs. 2d–f. The final RMSE on the test set was 7.56% for the conventional CNN and 7.55% with DE. The RMSE between the inputs and downsampled outputs for the conventional CNN was 0.10%.

e. NEXRAD vertically integrated liquid water

The SEVIR weather dataset (Veillette et al. 2020) is composed of collocated satellite and radar observations of 20 000 weather events over the continental United States between 2017 and 2020. Here we train EDRN to perform 4× SR on 192 × 192 pixel (1-km resolution) chips of vertically integrated liquid water (VIL), a radar derived product from the NEXRAD radar network. VIL has units of kilograms per meter squared, and the DE approach assures that the CNN is mass conserving; for example, the super-resolution process cannot inadvertently add or remove water that was not present in the input field. We use the 25 time samples from the first 18 000 events to train and the remaining 2000 to test. Figures 2g–i show an example case from the test set. The DE method achieves a slight performance improvement over conventional training (Fig. 3c). The final RMSE on the test set was 0.507 kg m⁻² for the conventional CNN and 0.502 kg m⁻² with DE. The RMSE between the inputs and downsampled outputs for the conventional CNN was 0.05 kg m⁻².

a. Neural networks

Here, we have reproduced seven different CNN architectures from the recent SISR literature. For unbiased intermodel comparison, we have altered each model slightly so that they all have a similar number of trainable parameters: 5 × 10⁶ (except for SR-CNN and Lap-SRN, which have fewer). The CNNs were implemented in Keras with a Tensorflow backend, and the code and model diagrams can be found on GitHub (https://github.com/avgeiss/invertible_sr). We provide a more detailed overview of the CNNs and our implementations in section 4 of the online supplemental material. The CNNs are SR-CNN (Dong et al. 2016), Laplacian Pyramid Super-Resolution Network (Lap-SRN; Lai et al. 2017), Dense U-Net (DUN; Ronneberger et al. 2015; Huang et al. 2017; Geiss and Hardin 2020), Deep Back Projection Network (DBPN; Haris et al. 2018), Dense SR Net (DSRN; Tong et al. 2017), EDRN (Lim et al. 2017), and Residual Dense Network (RDN; Zhang et al. 2018). Each was trained both with and without strictly enforced invertibility. We note that most of the examples of CNN-based SR applied to environmental datasets given in section 1e either directly use these CNN architectures or are close relatives of them.

b. Image datasets

The Div2k (Agustsson and Timofte 2017) dataset was used for training. It contains 800 high-resolution training images with a 100-image test set. The last 10 training images are typically held out and used to compute validation scores in SISR studies focused on development of new SISR models (Lim et al. 2017). Here, we do not perform hyperparameter tuning, but we do withhold these 10 images during training and use them to evaluate each model throughout the training process (Fig. 4). Trained CNNs are evaluated on several image test datasets that were used because of their prevalence in the SISR literature (Wang et al. 2020): “Set5” (Bevilacqua et al. 2012), “Set14” (Zeyde et al. 2010), “BSDS100” (Martin et al. 2001), “Manga109” (Fujimoto et al. 2016), “Urban 100” (Huang et al. 2015), and the 100-image “Div2k” test set (Agustsson and Timofte 2017). Manga 109 are illustrated images, and Urban 100 contains photographs of urban scenes, whereas the other datasets contain miscellaneous photographs.

View Full Size

(a)–(g) Div2k validation set PSNR during training for the seven CNN architectures with (red) and without (black) downsampling enforcement. The blue line in (e) uses an additional loss function term instead of DE. (h) Application of conventional CNN-SISR (black) and DE-SISR that incorrectly assumes 2D-average downsampling (green) to bicubic-downsampled images.

Citation: Artificial Intelligence for the Earth Systems 2, 1; 10.1175/AIES-D-21-0012.1

• Download Figure
• Download figure as PowerPoint slide

c. Training and testing

This section uses the exact training procedure described in section 3b. This training procedure and choice of optimizer parameters is very similar to both Lim et al. (2017) and Zhang et al. (2018). Two evaluation metrics are used for the image datasets: peak signal-to-noise ratio (PSNR; Wang et al. 2020) and the structural similarity index (SSIM; Wang et al. 2004). PSNR is computed on the intensity (Y) channel after converting the CNN’s output to the luminance (Y), chroma blue (Cb), and chroma red (Cr) (YCbCr) color space (Dong et al. 2016). SSIM is a scalar metric designed to be more representative of the perceptual quality of an image than pixelwise metrics. It scales from −1 to 1, and higher values are better. During model testing, each LR image is broken into 48 × 48 pixel chips using a 24-pixel stride, and PSNR and SSIM are then calculated on the 96 × 96 center portions of each of the HR outputs. For each CNN, both with and without DE, a five-member ensemble was trained from randomly initialized weights and the ensemble mean test scores are reported in Table 1. Figure 4 shows PSNR computed throughout training on the 10-image Div2k validation set for the first ensemble member for each CNN.

Table 1

Evaluation of several SR CNN architectures, both with (“w/DE”) and without downsampling enforcement, applied to standard test datasets for image SR. Entries show PSNR/SSIM, with higher scores in boldface type. Values are averaged across five training runs with random initializations, and italicized entries pass a 99% confidence test for difference in means.

View Table

d. Results

Figures 4a–g shows the Div2k validation set PSNR during training for a single CNN of each type (panels) both with (red lines) and without (black lines) the DE operator. All of the CNN architectures perform comparably or better when DE is added, with the largest advantage early during training.

Sample outputs from each of the CNNs for each of the training sets both with and without DE are included in the online supplemental material (Figs. 1 and 2 in the online supplemental material). While there are some differences on close inspection, the small differences in PSNR shown in Fig. 4 do not relate to any dramatic change in perceptual image quality of the output. These figures help demonstrate that the DE approach can achieve state-of-the-art SISR performance while strictly enforcing physical consistency within the CNN architecture.

Table 1 summarizes final performance for every CNN–test-set pair, with the better scores denoted by boldface text. Adding DE to the CNN improved performance in all but one case (the Dense-Net had slightly worse PSNR on Set5, although note that Set5 has only five images and that results in this column are more likely to be affected by small sample size). The improvements are often small but are of comparable size to recent generational improvements in CNN architectures—for example, EDRN versus RDN (https://paperswithcode.com/sota/image-super-resolution-on-bsd100-4x-upscaling; accessed 28 January 2021). Furthermore, in most cases the improvement in the mean test score due to adding DE passes a 99% confidence test (one-sided t test for difference in means; Rice 2006). These cases are shown in boldface italics. Overall, the results in Table 1 show that, in addition to achieving the primary goal of exact invertibility, our approach yields robust and consistent performance improvements when applied across a large sampling of the most prevalent CNN types and image datasets in the SISR literature.

a. Related methods

b. Other downsampling schemes

Assuming 2D-average downsampling is often correct for enforcing conservation laws, but it is not always used to train SISR-CNNs. We demonstrate the impact of an incorrectly assumed downsampling scheme by training the EDRN-CNN using a bicubic downsampling scheme, both with and without DE that assumes 2D-averaging. Predictably, the conventional EDRN outperforms the DE version in this case (Fig. 4h). For the div2k test set, the conventional scheme had a PSNR/SSIM of 30.26/0.8393, while the DE-EDRN scored 30.13/0.8356. Although 2D averaging is used here, it may be possible to derive similar operators for other downsampling schemes, by modifying (1) and (2) to include the weights of a bicubic downsampling kernel for instance.

c. Checkerboard artifacts

Some of the outputs from the SR-CNNs contain faint checkerboard artifacts (e.g., in the trees in the BSDS100 sample image in Figs. 1 and 2 in the online supplemental material). Because (1) imposes the low-resolution grid on the high-resolution CNN outputs, CNNs that use DE may be more prone to this “artifacting.” We perform a detailed analysis of checkerboard artifacts in section 5 of the online supplemental material and find that for the 4 × 4 SR used in this paper, the primary contributors to checkerboard artifacts are the use of an L2 instead of L1 loss and the use of 2D-average downsampling to prepare the training samples instead of the bicubic scheme. When larger-resolution increases (8 × 8) are performed the DE operator contributes substantially more, however. We note that in specific applications to scientific datasets, introduction of even very small discontinuities in the output fields could be physically problematic and that this is a limitation of our algorithm. Strategies for mitigation of this effect will be a useful area of future research.

d. The magnitude of pixel corrections

In section 2, f( $\mathsf{x}$, P) is interpreted as a correction applied to an intermediate image output by the CNN. Here, we investigate whether the quality of this intermediate output improves during training, or if the CNN learns to rely on the correction by examining the value of: | $\mathsf{x}$ − f( $\mathsf{x}$, P)|, the difference between the initial output and the corrected output. On the Div2k test set for the EDRN model, the average difference was 108 (when RGB pixel values are scaled to 0–255), meaning the HR image requires a correction to ensure that it will downsample to the input image. Figures 5a and 5b, show the intermediate output and the corrected output for an example HR image chip from the Div2k test set, respectively (the original image is shown in Fig. 5e). The magnitude of the correction can be reduced by retraining with a regularizer in the loss function

$\mathcal{L}=\text{MSE}+\lambda \overline{\left|\mathsf{x}-f(\mathsf{x},P)\right|},$(8)

where λ = 100. After training with this regularization term, | $\mathsf{x}$ − f( $\mathsf{x}$, P)| averaged over the Div2k test set was 0.3. This is demonstrated in Figs. 5c,d, where the intermediate output (Fig. 5c) is now a near perfect match for the final output (Fig. 5d). The overall performance of the SISR scheme was not substantially altered and the regularized CNN had a PSNR of 30.07 and a SSIM of 0.8359 on the Div2k test set, comparable to results without regularization. Because the final outputs are nearly identical with or without the regularizer, it is not necessary to include it for most SR use cases, but the ability to increase the accuracy of the CNN’s intermediate output may be useful for future applications.

View Full Size

Examination of intermediate outputs from our downsampling enforcement implementation of EDRN (Lim et al. 2017) for a sample image from BSDS100 (Martin et al. 2001) before and after the DE-operator is applied: (a) output prior to DE layer; (b) final output after DE layer; (c),(d) as in (a) and (b) but with regularization on the magnitude of the correction; and (e) the ground-truth image for reference.

Citation: Artificial Intelligence for the Earth Systems 2, 1; 10.1175/AIES-D-21-0012.1

• Download Figure
• Download figure as PowerPoint slide

e. Alternate formulations for other use cases

Equation (1) was formulated to enforce physical consistency and also enforce strict upper and lower bounds on the CNN outputs. Alternate and simpler formulations are possible if these bounds do not need to be enforced. In the case that only a lower bound needs to be enforced: data can be transformed to be greater than or equal to zero, the initial output from the CNN can be passed through a rectified linear unit transfer function, and then can be scaled by a constant to enforce physical consistency. This approach is applicable to downscaling atmospheric chemistry data (Geddes et al. 2016), where chemical concentrations are nonnegative, and upper bounds can be ignored for trace gases. If no strict bounds are required, a simple constant can be added to the initial CNN outputs to ensure physical consistency instead. This would be appropriate for super resolving wind vector data, for instance (Salameh et al. 2009).