a. Evaluation criteria

Forecast visualizations must satisfy several criteria to achieve their purpose. We propose the following four to evaluate the quality of photographic visualizations of weather forecasts:

1. Realism: The images should look real and be free of obvious artifacts. Ideally, it should not be possible to tell whether a given image was taken by an actual camera, or if it was synthesized by a visualization method.

2. Matching future conditions: The images should match the future atmospheric, ground, and illumination conditions in the view of the camera. However, matching every pixel of a future observed image is not possible, as the forecast does not uniquely determine the positions and shapes of clouds, for example.

3. Seamless transition: The visualization method should achieve a seamless transition from observation to forecast. It should reproduce the present image and must retain the present weather conditions as long as they persist into the future.

4. Visual continuity: The method should attain visual continuity between images for consecutive lead times. For example, ground and illumination conditions should not show unnatural changes between images.

b. Analog retrieval

For a fixed camera view, such visualizations could be created using analog retrieval from an annotated image database. Given an archive of past images from the camera, annotated with the weather conditions that were present at the time the picture was taken, the forecast could be visualized by retrieving the image that most closely matches the predicted conditions.

Analogs can be retrieved as individual images or sequences of images. Using per-image analog retrieval, ${\widehat{I}}_t^{\text{ind}}$ is the individual image from the archive where the associated weather descriptor ${\widehat{w}}_t$ is most similar to the forecast w_t for lead time t (e.g., as measured by Euclidean distance). As can be seen in the middle row of Fig. 2, using per-image analogs prioritizes matching the visible future weather conditions (our second evaluation criterion), but sacrifices the visual continuity between consecutive visualizations (our fourth criterion), with ground and illumination conditions changing abruptly between images.

View Full Size

(top) A sequence of images I₀, …, I₆, taken by the Flüela camera in the Swiss Alps between 1000 and 1600 UTC 2 Jul 2020. (middle) A nowcasting visualization created using analog retrieval of individual images from an annotated archive (see section 3 for a description of the data). Here, ${\widehat{I}}_t^{\text{ind}}$ is the individual image from the archive where the associated weather descriptor ${\widehat{w}}_t$ has the smallest Euclidean distance to the forecast w_t. (bottom) A visualization created using analog retrieval of image sequences. ${\widehat{I}}_0^{\text{seq}},\dots ,{\widehat{I}}_6^{\text{seq}}$ is the image sequence from the archive where the associated weather descriptors, concatenated as the vector $({\widehat{w}}_0,\dots ,{\widehat{w}}_6)$, have the smallest Euclidean distance to (w₀,…,w₆).

Citation: Artificial Intelligence for the Earth Systems 2, 1; 10.1175/AIES-D-22-0028.1

• Download Figure
• Download figure as PowerPoint slide

Using sequence analog retrieval, ${\widehat{I}}_0^{\text{seq}},\dots ,{\widehat{I}}_6^{\text{seq}}$ is the image sequence where the associated weather descriptors $({\widehat{w}}_0,\dots ,{\widehat{w}}_6)$ have the smallest distance to (w₀, …, w₆). As can be seen in the bottom row of Fig. 2, this method satisfies the fourth criterion by construction. However, to score well on the second criterion would require that the archive contains full sequences of images (and not just individual ones) that match the forecasted conditions. Consequently, sequence analogs poorly match the future atmospheric, ground, and illumination conditions, even if the archive contains a full year of past data (see Table 4 for our results).

Both analog methods satisfy the first criterion by construction, since the images were taken by the actual camera, but they score poorly on the third criterion (seamless transition from observation to forecast). Because webcam images jointly visualize many aspects of the visible weather, it is practically unlikely that the archive contains an image that exactly matches all of them, including the specific position and shape of clouds, ground conditions, illumination, etc. We found that one year of archive data was clearly insufficient (see section 4c for results), so one would have to collect multiple years of archive data to increase this likelihood, and therefore would have to wait this long before the method could be deployed. This severely limits the practical use of analog methods for our application.

c. Image synthesis

The choice of L is not obvious, however. Common regression losses, such as the L₂ or L₁ distance of pixel intensities, are not suitable for our application, since it is not possible to predict the exact location and shape of clouds from the weather forecast. Thus, a pixel-by-pixel equivalence of ${\widehat{I}}_t$ and I_t should not be sought. In fact, using a pixelwise loss function results in generated images that show a mostly uniform sky (see Fig. 3), unless G₀ is trained until it overfits and reproduces examples from the training data. Instead, L should measure how well ${\widehat{I}}_t$ matches the overall atmospheric, ground, and illumination conditions of I_t. That is, a human examiner should not be able to tell whether ${\widehat{I}}_t$ or I_t is the true camera image of the future, even though they will not be identical.

View Full Size

A pair of (left) generated and (right) real camera images, where G₀ was trained by minimizing the expected absolute difference of pixel values (the L₁ distance). While the ground and illumination conditions match quite well, the synthesized sky lacks detail because it is not possible to predict the exact location and shape of the clouds from the weather forecast.

Citation: Artificial Intelligence for the Earth Systems 2, 1; 10.1175/AIES-D-22-0028.1

• Download Figure
• Download figure as PowerPoint slide

d. GANs

To synthesize images that not only look realistic, but also correspond to the weather forecast at lead time t (our second evaluation criterion), w_t is provided as an additional input to both the generator and discriminator, G₂(z|w_t; θ) and D₂(I|w_t; η), in what is called conditional adversarial training (Mirza and Osindero 2014). See section 4b for details (in Table 3) of how well the synthesized images match the visible future weather conditions according to the criteria introduced there.

Our third goal is a seamless transition from the present image I₀ to generated future images ${\widehat{I}}_t$. For t = 0, the generator should therefore reproduce the present image, ${\widehat{I}}_0=I_0$. This can be achieved by extending the generator and discriminator once more. Instead of synthesizing images from a random input z, G₃(I₀, z|w₀, w_t) transforms the current image I₀ into ${\widehat{I}}_t$, based on the analysis state w₀ and the forecast w_t of the NWP model. z adds a random component to the transformation, enabling the generator to synthesize more than one image that is consistent with the forecast when t > 0 (see Fig. 15). The discriminator D₃(I|I₀, w₀, w_t) is also conditioned on the full input. For the rest of the paper, we drop the subscripts and refer to G₃ and D₃ as G and D.

GAN-based image transformation was introduced by Isola et al. (2017). The authors trained the networks using a linear combination of the regression objective in Eq. (1) and the adversarial objective in Eq. (2). The L₁ norm was used as the pixelwise regression loss, and the relative importance of the regression objective was set using a tuning parameter λ. We have found that in our application, setting λ > 0 sped up the training of G for synthesizing the ground but was detrimental for synthesizing clouds. Because their positions and shapes evolve over time, encouraging a pixelwise consistency was again not helpful and resulted in a sky lacking structure (as in the pure regression setting, Fig. 3). We therefore only use an adversarial objective for the training of G and D.

When the atmosphere is relatively stable on the scale of the lead time step size, there will be only small changes from I_t to I_t+1 in the position and shape of clouds. This should be reflected in the sequence of generated images ${\widehat{I}}_0,{\widehat{I}}_1,{\widehat{I}}_2,\dots$. The results in section 4d show that the generated sequences have a good degree of temporal continuity (the fourth criterion), even though we do not explicitly model the statistical dependency between consecutive lead times t and t + 1. Recurrent GANs (Mogren 2016) or adversarial transformers (Wu et al. 2020) are two potential avenues for further work in this area.

The pair of neural networks G and D can be trained for specific or arbitrary camera views. A view-independent generator could enable novel interactive applications, such as providing on-demand forecast visualizations for users equipped with a smartphone camera and a global navigation satellite system (GNSS) receiver. However, a view-independent generator will show its limits when the present view of the scenery is partially or fully blocked by opaque clouds or fog, and w_t predicts a better visibility in the future. Although G could generate natural looking scenery for the newly visible regions (Yu et al. 2018), it will not be the real scenery of this location, thus potentially confusing the user. We therefore only consider the view-dependent case in this paper, where the synthesized and real scenery match very well (see Fig. 11).

e. Related work

We are not aware of prior work that synthesizes web camera images to visualize weather forecasts. But GANs have and are being used as powerful tools for several related meteorological and climatological applications.

GANs have been used for the statistical downscaling and nowcasting of specific atmospheric fields, such as radar-measured precipitation (cf. Fig. 1, center) and cloud optical thickness. For example, Leinonen et al. (2021) and Price and Rasp (2022) developed stochastic superresolution GANs to generate ensembles of time-evolving high-resolution fields from low-resolution input fields. Ravuri et al. (2021) made use of adversarial training for probabilistic precipitation nowcasting, improving the forecast quality over previous deep learning approaches (e.g., Sønderby et al. 2020) by producing realistic and spatiotemporally consistent predictions, thus avoiding the problem of blurring and improving the skill on mid- to high-intensity rainfall.

GANs have also been used to transform images from one kind of visible weather to another. For example, Qu et al. (2019) used pix2pix for image dehazing, that is, improving the meteorological visibility by removing the effects of atmospheric scattering due to aerosols. Schmidt et al. (2019) developed a GAN to simulate the effects of flooding on user-provided images, visualizing the possible effects of catastrophic climate change. And Li et al. (2021) developed a cycle-consistent adversarial network (cf. Zhu et al. 2020) to transform images labeled as having either sunny, cloudy, foggy, rainy, or snowy weather. Transforming images from one distinct type of weather to another could in principle also be used for our task of forecast visualization. But describing the joint atmospheric, ground, and illumination conditions with a single discrete class label would result in a combinatorially large number of classes, rendering this approach impractical for our task. Recently, Requena-Mesa et al. (2021) proposed to synthesize satellite imagery conditioned on future weather and published a high-resolution dataset containing Sentinel-2 images, matching topography and meteorological variables. The three baseline models published by the authors do not include a GAN but incorporating adversarial training would be a natural extension of their Channel-U-Net model to possibly improve the forecast skill.

a. Network architecture

Both G and D are encoder–decoder networks with skip connections (see Fig. 4), similar to the U-Net of Ronneberger et al. (2015). At each stage ${\mathcal{E}}_s$ of the encoder, the layer height and width are halved and the number of channels (the depth) is doubled, while the inverse is done at each decoder stage ${\mathcal{D}}_s$. By concatenating the output of ${\mathcal{E}}_s$ to the output of the corresponding ${\mathcal{D}}_s$, long-range connections are introduced that skip the in-between stages.

View Full Size

The conceptual encoder–decoder architecture of the generator network $G:I_0,w_0,w_t,z\mapsto {\widehat{I}}_t$. Each encoder stage ${\mathcal{E}}_s$ halves the layer height and width and doubles the depth (blue dotted arrows), while each decoder stage ${\mathcal{D}}_s$ performs the inverse (orange dashed arrows). The output of ${\mathcal{E}}_s$ is also concatenated to the output of the corresponding ${\mathcal{D}}_s$ (green dot–dashed arrows), providing additional long range connections that skip the in-between stages. A transformation of the random input z is concatenated to the output of the innermost encoder stage ${\mathcal{E}}_5$. The complete architecture of G has additional pre- and postprocessing blocks and two more stages than shown in this figure. A full schematic of the network (including layer shape information) is available from the companion repository (https://zenodo.org/record/6962721/files/generator.png).

Citation: Artificial Intelligence for the Earth Systems 2, 1; 10.1175/AIES-D-22-0028.1

• Download Figure
• Download figure as PowerPoint slide

Each encoder stage ${\mathcal{E}}_s$ consists of three layers: a strided convolution (Conv) layer, followed by a batch normalization (BN) layer (Ioffe and Szegedy 2015) and a ReLU activation layer. The decoder stages have the same layer structure, except that a transposed convolution is used in the first layer.

A full specification and implementation of the networks is provided in the companion repository.² Here, we summarize the important architectural elements.

2) Discriminator

The input to D consists of the color channels of I₀ and I_t, and the weather descriptors w₀ and w_t, which are transformed into additional input channels as in G. The encoder and decoder again have five stages each, and there is a Conv-BN-ReLU preprocessing block before the first encoding stage and a postprocessing block after the last decoding stage.

The output of the discriminator has two heads to discriminate between real and generated images on the patch and on the pixel level (Schonfeld et al. 2020). The patch-level output D_p (see section 2b) is computed by combining the output channels of the last encoder stage using a 1 × 1 convolution. The pixel-level output D_ij is computed by a 1 × 1 convolution of the postprocessing output.

We tried alternative network architectures based on residual blocks (which add the block input to its output; see He et al. 2015), similar to the BigGAN architecture (Brock et al. 2019). While a generator using residual blocks learned to transform the ground faster than our final generator, it also created visible artifacts when transforming cloudy skies, see Fig. 5 for an example. We conjecture that residual blocks are less suited for transformations of objects that move or evolve their shape, because the generator has to learn a transformation that perfectly cancels them from the block input if they are not to appear in the output.

View Full Size

An example of visible artifacts introduced by a generator architecture that is based on residual blocks. Clouds in (left) the input image are still partially visible in the clear-sky regions of (right) the output image because the residual transformation learned by the generator does not fully cancel their appearance.

Citation: Artificial Intelligence for the Earth Systems 2, 1; 10.1175/AIES-D-22-0028.1

• Download Figure
• Download figure as PowerPoint slide

We also tried replacing the transposed convolution layers in the decoder stages with nearest-neighbor or bilinear interpolation followed by regular convolution, which was suggested by Odena et al. (2016) to avoid checkerboard artifacts. This kind of artifact was not noticeable in the output of our final generator architecture, while nearest-neighbor upsampling often produced axis-aligned cloud patterns, and bilinear upsampling often produced overly smooth clouds, see Fig. 6.

View Full Size

Examples of artifacts introduced by the upsampling method used in the generator. (left) Nearest-neighbor upsampling often produced axis-aligned cloud patterns, while (right) bilinear upsampling produced bloblike cloud shapes with a smooth boundary. Using transposed convolutions in the decoder stages avoided both problems.

Citation: Artificial Intelligence for the Earth Systems 2, 1; 10.1175/AIES-D-22-0028.1

• Download Figure
• Download figure as PowerPoint slide

Adding a second Conv-BN-ReLU block to each ${\mathcal{E}}_s$ and ${\mathcal{D}}_s$ also did not lead to a significant improvement, while doubling the number of network weights that had to be trained.

b. Training objective and optimizer

The training objective for G and D is an extension of Eq. (2). We omit the dependency on the trainable network weights θ and η in the following equations for the sake of brevity.

We evaluated three different optimization algorithms to train θ and η: stochastic gradient descent, rmsprop (Hinton et al. 2012), and Adam (Kingma and Ba 2017). Adam achieved the fastest improvement rate, but it could suffer from erratic spikes in the loss curve. Spectral normalization of the training weights (see section 2c) and small learning rates were necessary to achieve a smooth training progress.

We also evaluated multiple discriminator updates per generator update, and different learning rates for the training of G and D. We found that using a 2-times-faster learning rate for the discriminator (Heusel et al. 2017) achieved the best results and used 5 × 10⁻⁵ as the learning rate for the generator and 1 × 10⁻⁴ for the discriminator. The other Adam hyperparameters were set to β₁ = 0 and β₂ = 0.9.

c. Spectral normalization

GANs are difficult to train, because the optimization landscape of the adversarial training changes with every iteration. The loss can spike suddenly (after many iterations of consistent improvement), or the training can stall completely if the discriminator becomes too good at distinguishing real from generated images.

We use spectral normalization of trainable weights (Miyato et al. 2018) in all convolution layers to enforce Lipschitz continuity of the discriminator. As in Zhang et al. (2019), we found that using spectral normalization also in the generator further stabilizes the training.

a. Realism

To evaluate the realism of individual generated images, we asked five coworkers at MeteoSwiss (who regularly consult the cameras of the MeteoSwiss network for their job duties, but otherwise were not involved in this project) to examine whether a presented image looks realistic or artificially generated.

The evaluation data was generated in the following way. For every camera, 75 pairs (I₀, w₀) were sampled from the test data uniformly at random, but t = 0 was limited from 0600 to 1400 UTC, to avoid too many similarly looking nighttime views. Then a lead time t was sampled from 0 to 360 min (in increments of 10 min), and both the real future image I_t and the generated image ${\widehat{I}}_t=G(I_0,z|w_0,w_t)$ were added to the evaluation dataset. This sampling strategy ensured that there was no overall difference in the meteorological conditions of the real and generated images, which otherwise could have influenced the examiners’ accuracy.

Each examiner was then assigned 30 randomly selected images from each camera and asked to provide their judgment on the realism of each presented image. The examiners could take as much time as was necessary to come to a decision, consistent with the Human Eye Perceptual Evaluation Infinity (HYPE_∞) protocol proposed by Zhou et al. (2019) for the perceptual evaluation of generative models. The examiners were told that they would see real and generated images, but they did not know that the dataset was balanced, and neither did they keep a count of their judgments (i.e., to monitor whether their judgments were biased toward real or generated images). They could inspect the images at arbitrary zoom levels and also look at other images in the evaluation set before giving an answer. But they were not given any background information about the images, such as the date, the lead time of the forecast, or the predicted weather conditions.

The results of the evaluation are presented in Table 2. The overall accuracy of the examiners was 59% (corresponding to a HYPE_∞ score of 41%), with a 95% bootstrap confidence interval (CI) of 55%–64% (based on 10 000 bootstrap samples). The low accuracy of the examiners’ judgment indicates that it was challenging for them to distinguish between real and generated images (a 50% accuracy would correspond to random guessing). Many of the generated images look realistic enough to pass for a real image, but there were also instances where artifacts introduced by the generator were obvious at first glance (see Fig. 7 for an example).

Table 2

Results for the perceptual evaluation of real and generated images for the Cevio, Etziken, and Flüela cameras. Using a browser-based data labeling tool, images were presented to five examiners with the question “What is your first impression of this image?,” and they could answer either with “looks realistic” or “looks artificially generated.” Their answers were aggregated into a confusion matrix for each camera, where the rows correspond to the ground truth, and the columns correspond to the examiners’ judgment.

View Table

b. Matching future conditions

To evaluate how well the forecast visualization ${\widehat{I}}_t$ matches the future real image I_t, three examiners compared the atmospheric, ground, and illumination conditions visible in the images. As we do not expect ${\widehat{I}}_t$ to match I_t in every detail (e.g., the precise shape and position of clouds do not matter), we used the following descriptive criteria to determine their overall agreement:

• Atmospheric conditions

Cloud cover: clear sky | few | cloudy | overcast, cloud type: cumuliform | stratiform | stratocumuliform | cirriform, visibility: good | poor

• Ground conditions

Dry | wet | frost | snow

• Illumination

  Time of day: dawn | daylight | dusk | night, sunlight: diffuse | direct (casting shadows)

A mismatch between the actual and visualized conditions can happen because of two different kinds of failures. The forecast descriptor w_t can fail to accurately capture the weather conditions visible in I_t, or the generated image ${\widehat{I}}_t$ can be inconsistent with w_t:

1. 1) Inaccurate forecast: Besides COSMO-1 not having a perfect forecasting skill, evaluating the forecast fields only at the camera site can be insufficient to describe all of the visible weather. Furthermore, the spatial or temporal resolution of w_t can be too coarse for highly variable weather conditions. For example, the hourly granularity of the forecast can only approximately predict the onset of rain.
2. 2) Inconsistent visualization: The generator G can fail to properly account for the changes from w₀ to w_t in the transformation of I₀ to ${\widehat{I}}_t$.

The three examiners compared 50 pairs $(I_t,{\widehat{I}}_t)$ per camera (450 pairs in total), selected according to the same methodology as described in section 4a. For each pair, they determined whether the descriptive criteria match between the real and synthesized images I_t and ${\widehat{I}}_t$. To better understand if mismatches are due to either the first or second kind of failure, they also determined whether w_t accurately described all the weather conditions visible in I_t, and whether ${\widehat{I}}_t$ was consistent with w_t.

Table 3 summarizes the results of the evaluation. In general, cloud cover and cloud type were the most difficult criteria to get right. At Cevio, for example, in 44 out of 150 cases (29%) the generated amount of cloud cover differed from the observed future conditions. But in only 9 of those 44 cases was the mismatch due to the visualization method, meaning that w_t was accurate but ${\widehat{I}}_t$ was inconsistent with it (see Fig. 9 for a counterexample). The visualization method would therefore benefit from improving the accuracy of the weather forecast, for example, by using a downscaled version of the COSMO-1 forecast with a higher spatial and temporal resolution.

Table 3

Evaluation of the matching atmospheric, ground, and illumination conditions between real and synthesized images I_t and ${\widehat{I}}_t$. Three experts compared 50 randomly selected pairs $(I_t,{\widehat{I}}_t)$ per camera (450 pairs in total), according to the criteria specified in section 4b. They also examined whether the forecast w_t accurately describes the conditions visible in I_t, and whether ${\widehat{I}}_t$ is consistent with the forecast. All values are percentages of the possible maximum 150, and the values in parentheses are 95% bootstrap CIs. For values in bold font, the CIs do not overlap with the corresponding CIs for sequence analogs (see Table 4).

View Table

Table 4 shows the corresponding evaluation for sequence analogs (section 1b). Comparing the number of matching conditions for GAN based image synthesis and analog sequence retrieval, we find that image synthesis achieves significantly more matches for cloud cover, cloud type, ground conditions, and time of day. It also creates visualizations that are overall 27 percentage points more consistent with the weather forecast.

Table 4

Evaluation of matching weather conditions for sequence analogs ${\widehat{I}}_t^{\text{seq}}$ (see section 1b), following the same procedure as the evaluation of synthesized images in Table 3.

View Table

Figures 8–12 show further examples of the range of transformations that can be achieved by the generator, transforming the atmospheric and illumination conditions to match I_t, while retaining the specific ground conditions of I₀.

View Full Size

(top) A sequence of images taken by the Cevio camera between 1000 and 1600 UTC 6 Feb 2020, and (bottom) the corresponding images generated by G(I₀, z|w₀, w_t). This visualization satisfies our four evaluation criteria. The generated images look realistic and are free of artifacts. They match the real images of the future w.r.t. atmospheric, ground, and illumination conditions. The transition from observation to forecast is seamless: I₀ (at top left) is well approximated by ${\widehat{I}}_0$ (at bottom left), and ${\widehat{I}}_1$ retains the persisting weather conditions of I₀. Finally, there is good visual continuity, because the progression of shadows appears natural.

Citation: Artificial Intelligence for the Earth Systems 2, 1; 10.1175/AIES-D-22-0028.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

The cloud cover amount in (left) the generated image is too large, compared to (right) the observed conditions at Cevio on 1200 UTC 25 Apr 2020. But the mismatch in cloud cover is not a failure of the visualization method, as the COSMO-1 forecast predicts a 100% cloud area fraction in the medium troposphere.

Citation: Artificial Intelligence for the Earth Systems 2, 1; 10.1175/AIES-D-22-0028.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

(top) A sequence of images taken by the Etziken camera between 1400 and 2000 UTC 6 Feb 2020 and (bottom) the corresponding images generated by G(I₀, z|w₀, w_t). The generator was able to fully transform I₀ (at top left) from daylight to nighttime conditions, including the visual appearance of illuminated street lamps and windows at t = 4 h.

Citation: Artificial Intelligence for the Earth Systems 2, 1; 10.1175/AIES-D-22-0028.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

(top) A sequence of images taken by the Cevio camera between 1000 and 1600 UTC 17 Jun 2020 and (bottom) the corresponding images generated by G(I₀, z|w₀, w_t). The generator synthesizes the correct scenery (the Pizzo Paràula mountain in the center of the image) that is occluded at t = 0 and becomes visible at t = 5 h.

Citation: Artificial Intelligence for the Earth Systems 2, 1; 10.1175/AIES-D-22-0028.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

(top) The same sequence of images as in the top row of Fig. 2, taken by the Flüela camera between 1000 and 1600 UTC 2 Jul 2020. (bottom) Because ${\widehat{I}}_t=G(I_0,z|w_0,w_t)$ is a transformation of I₀, the exact position and shape of snow patches are retained in ${\widehat{I}}_1,{\widehat{I}}_2,\dots ,{\widehat{I}}_6$.

Citation: Artificial Intelligence for the Earth Systems 2, 1; 10.1175/AIES-D-22-0028.1

• Download Figure
• Download figure as PowerPoint slide

c. Seamless transition

For a seamless transition between observation and forecast, the generator must be able to reproduce the input image ${\widehat{I}}_0=I_0$ for t = 0. It must also retain the conditions of I₀ in ${\widehat{I}}_1,{\widehat{I}}_2,\dots$ as long as they persist into the future.

We evaluated the third criterion on hourly nowcasts up to six hours into the future. As can be seen in Figs. 8–13, the generator reproduces the ground and illumination conditions of I₀ very well in ${\widehat{I}}_0$. The shape and positions of clouds are reproduced closely but not exactly, giving the impression that the generator reconstructs the overall cloud pattern but not every small detail. The resulting pixelwise RMSE between the synthesized and real images ${\widehat{I}}_0$ and I₀ is on average 4 times lower than for sequence analogs ${\widehat{I}}_0^{\text{seq}}$: it is 1.11 × 10⁻² versus 4.64 × 10⁻², computed on 1000 pairs $({\widehat{I}}_0,I_0)$ per camera. We could achieve pixel-level accurate reproductions of I₀ by using a residual architecture for the generator network. But as discussed in section 2a, the realism of images generated by a residual network suffered from noticeable visual artifacts for t > 0.

View Full Size

(top) A sequence of images taken by the Flüela camera between 0600 and 1200 UTC 12 Jan 2020. (bottom) The generator accurately transformed the illumination conditions and shadows, but it failed to learn how the sun moves across the sky. Instead of shifting the position of the sun, the sun stays in the same position and fades away gradually.

Citation: Artificial Intelligence for the Earth Systems 2, 1; 10.1175/AIES-D-22-0028.1

• Download Figure
• Download figure as PowerPoint slide

As can be seen in the bottom row of Fig. 12, the generator retains the specific conditions of I₀ (such as the position and shape of snow patches) in the visualizations ${\widehat{I}}_1,\dots ,{\widehat{I}}_6$. This is only possible because G transforms I₀ into ${\widehat{I}}_t$. As already shown in the corresponding Fig. 2, achieving a seamless transition is not feasible using analog retrieval (section 1b), because an image with the specific shapes and positions of clouds and snow patches is unlikely to exist in the archive.

d. Visual continuity

Finally, the consecutive visualizations ${\widehat{I}}_t,{\widehat{I}}_{t+1}$ must show visual continuity, with a natural looking cloud development, change of daylight, movement of shadows, and so on. As can be seen in Figs. 8–14, the evolution of the ground and illumination conditions matches the future observed images very closely, even though G and D do not explicitly account for the statistical dependency between t and t + 1. The increase or decrease in cloud cover and visibility also looks natural. By contrast, the visualizations based on individual analogs ${\widehat{I}}_t^{\text{ind}}$ lack continuity (Fig. 2, middle row), with snow patches appearing and disappearing, and illumination conditions changing unnaturally.

View Full Size

(top) A sequence of images taken by the Cevio camera between 1000 and 1600 UTC 17 Feb 2020. (bottom) The generated images accurately match the ground and illumination conditions, and the development of cloud shapes appears natural. But the positions of individual clouds remain too static over time.

Citation: Artificial Intelligence for the Earth Systems 2, 1; 10.1175/AIES-D-22-0028.1

• Download Figure
• Download figure as PowerPoint slide

But the generator struggled with learning image transformations that involve translations of objects across the camera view, such as the movement of the sun (Fig. 13) or of isolated clouds (Fig. 14). We conjecture that a network architecture based on Conv-BN-ReLU blocks is highly effective at transforming the appearance of objects that remain in place, but less so for translation operations. We therefore investigated whether including nonlocal self-attention layers (Zhang et al. 2019) could be beneficial. We did not achieve a consistent improvement in our experiments, neither with full nor with axis-aligned self-attention, while the network training time increased significantly.

The balance between visual continuity and image diversity can be tuned by changing the standard deviation σ of the random input $z_i\sim \mathcal{N}(0,{\sigma }^2)$. Increasing σ leads to a greater diversity of visualizations that are deemed consistent with the weather forecast, while decreasing σ promotes greater continuity between subsequent images (see Fig. 15). We have found that setting σ = 0.5 results in the best possible trade-off between the two objectives.

View Full Size

The effect of the noise variance on the diversity of visualizations synthesized by the generator. Increasing σ when sampling $z_i\sim \mathcal{N}(0,{\sigma }^2)$ leads to a greater diversity of images that are deemed consistent with the weather forecast. But note that the realized diversity is also a function of t: it is smallest at t = 0, and greatest when there is a change of weather conditions at t = 3 h. (top) A sequence of images taken by the Cevio camera between 0600 and 1200 UTC 16 Mar 2020. (bottom rows) Nowcasting visualizations generated with σ = (0, 0.2, 0.5, 1.0).

Citation: Artificial Intelligence for the Earth Systems 2, 1; 10.1175/AIES-D-22-0028.1

• Download Figure
• Download figure as PowerPoint slide