a. Unlabeled MODIS imagery

MODIS is a multispectral imager that orbits aboard NASA’s EOS Terra satellite (Salomonson et al. 1989, 2006). Since 1999, it has collected data in a polar, sun-synchronous orbit, with an equatorial crossing time of 1030 local time (LT) and has now produced a multidecade single instrument dataset of high-quality images of Earth. Here, we have constructed a dataset of Terra MODIS images for training the self-supervised model. Specifically, we use the 500-m Level-1b radiance product (file name: MOD02HKM; MODIS Characterization Support Team 2017), and convert the radiances to reflectances. Reflectances are used because neural networks generally perform better with bounded and/or standardized data. Reflectance is naturally bounded to the range [0, 1] and is a physically meaningful value. The dataset includes a full year (2020) of MODIS images taken during daytime over the world’s oceans. We use data from band 4 (550 nm), which is ideal for daytime visible wavelength observations of clouds. It is also used for aerosol and land surface retrievals and is the green band for generating true-color [red–green–blue (RGB)] MODIS imagery (Salomonson et al. 2006). MODIS uses an internal mirror to scan perpendicular to its orbit and has a resulting swath width of about 2330 km. Some distortion caused by the along-orbit motion of the satellite and changing pixel footprint while scanning (the “bowtie” effect), can occur near the edges of the swath (Gómez-Landesa et al. 2004). Here, we opt to crop the MODIS images to the center 1024 pixels (of 2708 for the 500-m product) where the distortion is negligible, which limits the images in the dataset to a maximum viewing zenith angle of about 21°. The dataset is further limited to granules that have no more than 10% of their pixels flagged as land or sub-50-m-deep ocean, and to granules with average solar zenith angle less than 60° to ensure sufficient scene illumination. Finally, we break up the cropped granules along swath into 1024 × 1024 pixel images that are a more convenient size for our ML algorithm to work with. These training images are further subsampled during the training process to generate 256 × 256 pixel image pairs used by the CNN, as described in section 3c, but the training dataset is stored on disk as a collection of 64 340 unique 1024 × 1024 images.

b. Unlabeled ABI imagery

The Advanced Baseline Imager (ABI) (Schmit et al. 2017) is a multispectral, 16-band imager aboard the GOES-17 satellite, which was launched in 2018 and occupies a geostationary orbit above 137.2°W at the equator. Here, we use reflectance images from the ABI’s 2-band that has a central wavelength of 640 nm, an approximate pixel resolution of 500 m at nadir, and is well suited for observing daytime clouds. The ABI operates in several scan modes that are changed based on imaging needs, but currently (since 2019) collects a “full disk” scan approximately every 10 min (Schmit et al. 2017). Here, to sample a diversity of cloud scenes, we have constructed a cloud image dataset by acquiring one full-disk scan for each day of 2021 and have selected the scan taken closest to solar noon for 137.2°W, which ensures that the majority of the scene is illuminated. The year 2021 was chosen to avoid overlap with the MODIS dataset. Similar to the MODIS dataset, the full disk scans were broken into nonoverlapping 1024 × 1024 pixel images, and we enforce requirements that each image has an average solar zenith angle less than 60° with less than 10% land pixels. These constraints result in 91 077 total images.

c. Labeled imagery from Yuan et al. (2020)

Yuan et al. (2020) constructed an expert labeled cloud imagery dataset using 1-km resolution Aqua MODIS channel 4 (550 nm) observations broken into 128 × 128 pixel image chips. Their study focuses on marine low-cloud morphology, so they filtered the image dataset to exclude scenes with more than 10% high cloud, less than 5% low cloud, or more than 10% land coverage. They asked a group of experts to separate the cloud imagery into six categories to create a dataset of hundreds of labeled cloud images (the numbers in parentheses below represent the number of labeled samples from each category used in this study):

• Stratus: (455) A mostly uniform layer of low cloud.

• Closed cellular: (428) A cellular structure with cloudy centers that frequently organizes into a honeycomb pattern.

• Disorganized cellular: (427) A combination of strong convective elements surrounded by stratiform cloud.

• Open cellular: (468) Spatial cells with a clear center and vigorous convection at the edges.

• Clustered cumulus: (368) Aggregate convective cloud elements with relatively strong convection.

• Suppressed cumulus: (363) Multiple small disperse cumulus clouds.

They then used a pretrained CNN [Visual Geometry Group, model 16 (VGG-16); Simonyan and Zisserman 2014] to demonstrate automated classification of cloud morphology, achieving a 93% categorical accuracy on a held-out testing set. They demonstrated the use of the trained CNN to study the frequency of occurrence of the cloud regimes for winter 2011 in the North Atlantic and to generate a 2003–18 cloud climatology for the southeast Pacific.

The authors of Yuan et al. (2020) have provided a dataset of 2509 hand-generated annotations of Aqua MODIS images from 2010 and 2020 for use in this study (these are not identical to the ones used in their study). Their original samples are 128 × 128 pixel images of 1-km 550-nm reflectance, and we have retrieved 256 × 256 pixel versions at 500-m resolution for consistency with our unlabeled MODIS training data.

d. Labeled imagery from Rasp et al. (2020)

Stevens et al. (2020) performed an in-depth study of cloud regimes that occur in the North Atlantic trade wind region during the winter (December–February). They used visual inspection of MODIS L1B corrected reflectance imagery to identify four distinct and common mesoscale cloud structures:

• Sugar: Disorganized fine-scale clouds. Primarily composed of cumulus humilis.

• Gravel: Larger boundary layer cumulus features, often precipitating with deeper development than sugar.

• Fish: Organized regions of boundary layer cumulus with a skeletal structure separated from other clouds by large clear regions.

• Flowers: Large organized stratiform cloud regions with a precipitating cumulus core. Typically, multiple distinct neighboring clouds occur at the same time.

Their choice of these four categories was based on their frequency of occurrence and the ability of independent labelers to consistently identify them. Ultimately, 12 experts hand-labeled 10 winter seasons worth of data (900 images) using images that covered 10° × 10° regions. Stevens et al. (2020) used the dataset to analyze differences between labelers and rate of class confusion and provided some additional exposition on the 3D cloud structure of each category using radar observations. Rasp et al. (2020) later extended their work by using crowd-sourced labeling to significantly increase the size of the labeled cloud imagery dataset, from which they trained and evaluated a CNN to perform the same classification task. They used Terra and Aqua MODIS cloud imagery from two regions in the subtropical Pacific and one in the subtropical Atlantic observed between 2007 and 2017. Their labels are human-drawn rectangular bounding boxes that contain primarily a single cloud type from one of the categories above. They estimate approximately 250 person-hours were required to generate 49 000 unique labels.

The neural networks trained here, and the majority of self-supervised image processing schemes, are designed to operate on image chips. Here “image chips” refers to 256 × 256 pixel samples of a satellite image. Because the dataset produced by Rasp et al. (2020) is structured as labeled bounding boxes applied to large MODIS images, using it for evaluation of our models required reprocessing the original labeled images to produce a new dataset of labeled image chips. On inspection of the bounding boxes, it is clear that labelers often draw large bounding boxes that may contain a cloud type of interest but are also likely to contain large regions of clear sky or disorganized cloud that does not clearly match one of the class definitions. Furthermore, multiple labelers were asked to label the same images and they often produced overlapping bounding boxes with different labels or with similar labels but significant disagreement on the spatial extent of the bounding box (the inconsistency of human labelers is another drawback of supervised methods when class boundaries are not well defined). To generate an evaluation dataset with consistent samples, we extracted all regions where at least three labelers agreed on the cloud category from the 9904 images used by Rasp et al. (2020). These were then split into 256 × 256 pixel chunks with no overlap. Because the classes are imbalanced, with significantly more samples of “gravel” than “flowers” for instance, we retained only 2200 samples from each category (corresponding to the class with the fewest samples) resulting in an evaluation dataset of 8800 image chips. Because of the number of steps required to generate this dataset, we have made it publicly available (see the data availability statement below). Throughout the remainder of this article, we use the acronym of sugar, gravel, flower, fish (SGFF) to refer to this dataset.

a. Encoder

We use a ResNet-style (He et al. 2016) CNN as the encoder. This is a common and powerful architecture for image processing tasks and is the architecture used by Zbontar et al. (2021). ResNet is notable for including a large number of skip connections that allow the network to easily learn representations across multiple resolutions and combat the vanishing gradient problem (Goodfellow et al. 2016). Here, we have reimplemented the ResNet instead of replicating one of the architectures proposed by He et al. (2016), but use the same key building blocks. There are two large differences in our implementation: we have opted to use 128 km (256 pixel) image chips for consistency with past mesoscale cloud morphology studies (Wood and Hartmann 2006; Yuan et al. 2020) instead of the 224 × 224 pixel inputs that are typically used for training on ImageNet (Deng et al. 2009) and have modified the architecture to accommodate this. We also use a smaller latent space and output a 1024-dimensional vector whereas ResNet-50 outputs 2048 features from its CNN component (before applying a classifier head). The exact implementation details can be found in the project’s code repository and a diagram of both the encoder and projector is provided in Fig. A1 in the appendix. Overall, the encoder CNN has 18.6 million trainable parameters.

b. Projector

The projector network is a 3-layer fully connected neural network that maps the vector outputs of the ResNet encoder to a higher dimensional space where the loss function is computed. Here, we are using a smaller image embedding space than Zbontar et al. (2021) and opted to use a smaller projector network than they did that has progressively increasing layer sizes of 2048, 4096, and 8192 (also shown in Fig. A1). The lower number of trainable parameters reduces the computational requirements of training; meanwhile, the output from the projector network remains high dimensional, which has been observed to improve model skill. We hypothesize that a slightly lower complexity model will be adequate to generate useful representations (vector embeddings) of the image data because cloud imagery likely contains less diverse content than the assorted photographs used in ImageNet (Deng et al. 2009), though additional experimentation would be required to verify this. Nonetheless, this projector configuration has 44 million trainable parameters and is substantially larger than the encoder, which is a common feature of Barlow Twins and related self-supervised schemes (Bardes et al. 2021).

c. Image augmentations

Reliance on image augmentations is a critical aspect of most modern self-supervised schemes (Gidaris et al. 2018; T. Chen et al. 2020; Bardes et al. 2021; Grill et al. 2020; Chen and He 2020). Performing image comparisons is an effective and intuitive way to learn deep feature representations of unlabeled image data, but augmentations must be used to avoid learning trivial solutions (comparing bulk image statistics like image color, brightness, or contrast for instance) and drive the models to distill semantically meaningful image representations. This work uses a procedure for generating image pairs during training that accounts for the unique properties of satellite cloud imagery. First, the unlabeled imagery described in sections 2a and 2b, has been sampled to nonoverlapping 1024 × 1024 pixel scenes. The semantic content in these scenes is often spatially autocorrelated, meaning that neighboring regions in the 1024 × 1024 pixel training images tend to contain the same cloud types. We leverage this when generating training samples by randomly selecting paired image chips from two locations in the same 1024 × 1024 pixel region. Second, single cloud types can vary drastically in scale. Stevens et al. (2020) notes that the potential spatial extent of individual cloud elements in their flowers category can span scales from 20 to 200 km for instance. When sampling image pairs, we randomly choose image chips between 128 × 128 pixels and 512 × 512 pixels in size and rescale them to 256 × 256 pixels using bicubic interpolation. Third, image flips and rotations do not change the cloud type observed, so samples are randomly flipped horizontally and vertically, and randomly rotated by integer multiples of 90°. Last, scene illumination can change drastically depending on viewing and solar zenith angles, and the same cloud morphology can occur under a variety of different lighting conditions. We leverage this by randomly altering the image brightness and contrast by factors of 0.5–1.5 during training. This specific procedure was chosen based on our knowledge of the dataset and these image transforms can be applied safely without changing the cloud type present in an image. While we did not perform extensive experiments to determine the relative value of individual image augmentations like T. Chen et al. (2020), we believe that these are reasonable choices based on domain knowledge and show in section 4 that they were effective for training the CNN.

d. Training procedure

a. Evaluation procedure

Quantitative evaluation of the quality of a self-supervised image vector embedding space is not particularly straightforward because there is no ground-truth for comparison. We follow an approach that has been frequently deployed in the ML literature of training a simple linear classifier model to ingest vector embeddings created by the self-supervised encoder CNN and produce image labels for the labeled image datasets introduced in sections 2c and 2d (van den Oord et al. 2018; T. Chen et al. 2020; Chen and He 2020; Misra and van der Maaten 2019). For large image datasets, a simple linear classifier does not have the capacity to directly classify image pixel data. Therefore, if a self-supervised CNN can learn representations of an unlabeled image dataset that a linear classifier can leverage to achieve high classification accuracy on a labeled dataset, then the CNN must have learned useful feature representations of the images.

The linear model is a single layer neural network with a softmax output (also commonly known as a linear classifier or logistic regression). It outputs a probability for each cloud category in the corresponding labeled imagery dataset and has one input for each value in the image embedding space, meaning it has 1024 × C trainable weights and C biases where C is either 4 or 6 depending on the dataset used. We train it by concatenating it to the trained encoder CNNs after freezing their weights. This means that while the encoder CNNs are included as part of these models, only the linear classifiers’ weights can be updated when training on the labeled imagery. Because of the relatively small size of the labeled datasets, we apply a 20% dropout to the linear classifier’s input during training to prevent overfitting and use random horizontal and vertical image flips to augment the labeled cloud image datasets at this stage. We use a 70%, 10%, 20% train/validation/test split of the labeled imagery datasets. The classifiers are trained while monitoring the validation loss with an early-stopping callback using a patience of 10 epochs. This is done once with an initial learning rate of 10⁻³, and then again after lowering the learning rate by a factor of 10. We use the Adam optimizer (Kingma and Ba 2014) with the Keras (Chollet et al. 2015) default settings of β₁ = 0.9 and β₂ = 0.999 and a batch size of 32. The test-set accuracy of these linear classifiers on each category in the labeled imagery datasets is shown in Tables 1 and 2, the overall categorical accuracy is compared to other benchmark models (introduced in the following sections) in Table 3, and the results are discussed in more detail in section 4d.

Table 1.

Confusion matrix for a simple linear classifier model trained to recognize the sugar, gravel, fish, and flowers cloud types using vector embeddings from the self-supervised model. Bold values are true positives.

View Table

Table 2.

Confusion matrix for a simple linear classifier model trained to recognize the cloud types from Yuan et al. (2020) using vector embeddings from the self-supervised model. Bold values are true positives.

View Table

Table 3.

Categorical accuracy for linear classifiers trained on top of several different convolutional neural networks for two different labeled cloud imagery datasets. Two self-supervised models are shown, one trained on ABI imagery and one trained on MODIS imagery. DenseNet-121 and ResNet-50 are CNNs that have been pretrained on the ImageNet dataset, and the linear classifier is applied to the features from their last convolutional layer (after applying global average pooling). ResNet-CNN is a ResNet-style CNN that is trained from scratch with a densely connected layer identical to the linear classifier as the output layer.

View Table

b. Benchmark models

To provide context for the accuracy of the self-supervised classifier approach (described above), we compare against several benchmark models. Two types of benchmarks are considered: training a full CNN directly on the labeled imagery datasets (“from scratch”) and training a linear classifier on the image embeddings from two different types of CNNs that were pretrained in a supervised manner on ImageNet (“pretrained”). The categorical accuracy of the self-supervised classifiers is compared to these benchmark models in Table 3 and discussed in greater detail in section 4d.

The “from scratch” benchmark can be described as follows: The traditional approach to using a CNN for a classification task is to train the CNN (with a linear classifier as its last layer) solely on a labeled dataset. For this benchmark, the same type of linear classifier head is appended to a CNN with randomly initialized trainable weights and the exact same architecture as the self-supervised encoder CNN (Fig. A1). The whole CNN and classifier head are trained directly on each of the small labeled imagery datasets using the same procedure and train/validation/test split described in the section above. Comparing the skill of the classifiers where the CNN is trained on the much larger unlabeled imagery dataset and only the linear classifier is trained on the labeled dataset to the full CNNs trained completely on the labeled dataset gives an estimate of both the value of self-supervised transfer learning and the relative quality of the deep representations learned in the self-supervised training process. These trained from scratch models generally perform much worse than the self-supervised models, though could likely perform comparably with a substantially larger labeled training dataset.

The “pretrained” benchmark can be described as follows: A common approach when facing limited labeled training data is using a CNN that has been pretrained on a very large labeled imagery dataset in a supervised manner and fine-tuning the weights of the classification layers at the end of the model to the target task [this technique was used by Yuan et al. (2020)]. Here, we follow a similar setup to the one used for our self-supervised models. We train a linear classifier applied to a global average pooling of the output from the final convolutional layer of ResNet-50 (He et al. 2016) pretrained on ImageNet to classify the labeled cloud imagery datasets. Resnet-50 has a 2048-value embedding space, so we also compare to DenseNet-121 (Huang et al. 2018), which has the same embedding dimensionality as our model. The pretrained weights for these two CNNs are distributed as part of Keras (Chollet et al. 2015). These two CNNs were not originally trained on 256 × 256 pixel imagery so a resizing layer is used to convert the images to their native input resolution (224 × 224 pixels). This experiment helps characterize the relative value of training on cloud imagery versus a largely unrelated natural image dataset, assessing whether the self-supervised approach can learn representations of comparable quality to those learned by a supervised CNN trained on a large labeled dataset.

Last, an identical self-supervised model was also trained on a library of ABI imagery, which is discussed in greater detail below in section 4c. All of the benchmark models were trained using the same approach (early stopping, optimizer, learning rate, etc.) as the combined self-supervised and linear classifier model.

c. Cross-instrument application

The image augmentations used while training the self-supervised CNN prevent it from learning trivial ways to minimize the Barlow Twins loss function, but they also encourage resilience against these types of transformations in downstream applications. The image resizing, flips, rotations, and brightness adjustments used during training make the CNN’s embeddings robust against external sources of similar variability, such as small differences in resolution or spatial distortions from a different satellite imager, different spatial orientations of cloud patterns in different geographic regions and weather patterns, or different scene illumination. Most algorithms underlying satellite retrievals are bespoke, and direct cross-instrument application is not possible, so the prospect of an algorithm that can be trained using one instrument and applied to others (or potentially trained on a multi-instrument dataset) is particularly novel. In this experiment, we test whether the feature representations learned by a self-supervised CNN from ABI images can be leveraged by a linear classifier to accurately identify cloud in images from MODIS, two distinct imagers in different orbits.

The most straightforward way to test this concept would have been to evaluate the self-supervised CNN that has already been trained on MODIS imagery on its ability to accurately label a hand-labeled ABI cloud imagery dataset, but the only hand-labeled images available for testing are from MODIS (sections 2c and 2d). Instead, a second copy of the self-supervised model was trained using the unlabeled ABI imagery dataset (section 2b) following the same training procedure described in section 3d. Table 3 also shows the same linear classifier evaluation procedure discussed in section 4a applied to image embeddings of the MODIS evaluation datasets produced by the CNN trained only on ABI imagery (in the columns labeled “self-supervised” and “ABI”).

d. Classification skill

Tables 1 and 2 show confusion matrices for the linear classifier trained on the self-supervised CNN’s embeddings of the SGFF (Rasp et al. 2020) and the Yuan et al. (2020) datasets, respectively. The lowest and highest skill for each dataset occurs in the fish and sugar SGFF categories, respectively, and “clustered cumulus” and “open cellular” Yuan et al. (2020) categories, respectively. Both linear classifiers achieve skill much higher than random guessing for all cloud categories. From this we can infer that the vector embeddings generated by the self-supervised algorithm successfully encode information about mesoscale cloud structure.

Table 3 shows the test-split categorical accuracy for both labeled datasets across the various classifier model types. In the remainder of this paragraph, we have italicized the words that directly correspond to columns and subcolumns in the table. As a short summary, this table includes results for two trained from scratch CNNs, one trained on each of the evaluation datasets; two self-supervised CNNs, trained on each of the unlabeled imagery datasets, each with a separate linear classifier trained on each of the evaluation datasets; and two CNNs pretrained on ImageNet, each with a separate linear classifier trained for each evaluation dataset. For SGFF (Rasp et al. 2020), the larger of the two labeled imagery datasets, both self-supervised models, with CNNs trained on either the unlabeled MODIS or ABI imagery datasets, outperform the CNNs that were pretrained on ImageNet and the trained from scratch CNNs by several percent. The self-supervised model trained on MODIS imagery performs slightly better than the one trained on ABI imagery, which is expected because the labeled image datasets contain MODIS imagery. For the smaller Yuan et al. (2020) dataset, the ResNet-50 pretrained model performs the best with the MODIS- and ABI-trained self-supervised models performing second and third best, respectively. However, both self-supervised models outperform the DenseNet-121 pretrained model, which has the same size embedding space. The trained from scratch CNN underperforms the others by a wider margin for this dataset, likely because of the smaller training-split size and the need to use early stopping to prevent overfit. Our expectation is that the trained from scratch CNN would perform comparably to the other models if we had access to significantly more labeled training data but performs much worse in this data-limited setting.

The categorical accuracy of the self-supervised model trained with ABI images is only slightly lower than the model trained with MODIS images: 0.6% for SGFF and 1.9% for the Yuan et al. (2020) dataset. This indicates that the self-supervised CNN embeddings are relatively robust against changes in imager properties. Importantly, a linear classifier leveraging embeddings from the self-supervised CNN that was trained on unlabeled ABI data performs better than a CNN trained directly on the labeled MODIS datasets, demonstrating this method’s potential for cross-instrument pretraining in data-limited scenarios. While here we have demonstrated a cross-instrument application between two contemporary instruments, this result suggests that applying the algorithm across multiple generations of instruments should also be possible. For instance, the ABI’s 640-nm band was carried over from the past generation of ABI imagers, and the Visible Infrared Imaging Radiometer Suite (VIIRS), which will carry on some of MODIS’s functionality when it deorbits (Platnick et al. 2021), includes 550- and 640-nm bands at 675- and 375-m resolution, respectively. An instrument-agnostic algorithm also suggests the possibility of multi-instrument studies of cloud morphology and climatology, without the same degree of concern about introducing cross-instrument artifacts, a common frustration for past multi-instrument cloud studies (Rossow and Schiffer 1999).

Overall, these results demonstrate the ability of self-supervised CNNs to learn deep representations of cloud images that efficiently encode information about the mesoscale cloud morphology. For the purposes of future cloud studies that focus on cloud structure, and particularly those that might have limited labeled data to work with, pretrained models using self-supervision will be a more powerful tool than classifiers trained directly on the available labeled data or ImageNet pretrained CNNs. This approach, catalyzed by the abundance of historical cloud imagery, permits future cloud studies that were previously impossible or prohibitively labor intensive.

a. Comparisons in embedding space

b. Detection and search with limited labels

A valuable use case for the vector representations is performing image search and classification with only one, or perhaps a handful, of labeled images. This can be done via comparisons of images in the embedding space under the assumption that nearby embeddings will correspond to similar cloud morphologies. In Figs. 1 and 2, examples are shown of a single query image from each cloud category in the two labeled imagery datasets followed by the closest 5 images drawn from the unlabeled MODIS dataset. While we did not enforce this constraint, no two images of the same cloud type in Figs. 1 and 2 were captured on the same day, with the exception of two of the flowers images in Fig. 1, and even those were drawn from different MODIS granules. By examination, the cosine similarity comparison has identified new samples that contain similar cloud morphologies to each of the query images and has identified unique cases from throughout the imagery dataset.

For a more quantitative evaluation of embedding-space comparisons, we can calculate the classification skill of a k-nearest-neighbor algorithm applied to the vector embeddings of each of the labeled imagery datasets (Bishop 2006). Here, we chose k = 40 by evaluating a k-nearest-neighbor classifier on the validation splits using various values of k. Then, for each image in the testing split, we assign the most common label among the 40 closest (by cosine distance) images in the training split. This approach achieves a categorical accuracy of 84.4% on the SGFF dataset and 73.1% on the Yuan et al. (2020) dataset. These accuracies are high considering the simplicity of the k-nearest-neighbor model and approach the skill of the linear classifiers trained in section 4. This result further verifies the quality of the embedding space for comparing different cloud morphologies.

c. Visualizing learned representations

CNNs’ internal representations are difficult to interpret. Direct examination of model weights, for instance, does not provide much information about what CNNs learn except in the early layers of a network, so numerous indirect interrogation techniques have been developed (Qin et al. 2018; Nguyen et al. 2019). Here, we apply “activation maximization” (Erhan et al. 2009; Simonyan and Zisserman 2014) to visualize the variety of features learned from cloud imagery.

Activation maximization leverages the automatic differentiation capabilities of ML software libraries (Chollet et al. 2015). The process involves selecting a single activation from the encoder network to interrogate, generating an initial image with random pixel intensities, and then iteratively updating the image’s pixel values to maximize the value output by the target activation. Our application of activation maximization seeks to identify what has been learned by the output neurons from the encoder network. In other words, what cloud morphologies are represented by each dimension of the embedding space? We used 2500 iterations of stochastic gradient descent to iteratively push white-noise images toward pixel values that excited each output neuron from the trained encoder and found that the process resulted in a wide variety of distinct spatial patterns. Figure 3a shows 12 example cases. While there is a great deal of variety in the activation maximization images, we note that some output neurons produce qualitatively similar patterns to others, perhaps implying that the Barlow Twins algorithm has not entirely eliminated redundancies in the embedding space or that a smaller embedding space would have been sufficient. Nonetheless, the visualizations show that the CNN has internal representations of a range of distinct cloud structures.

View Full Size

(a) Activation maximization applied to a selection of neurons in the output layer of the encoder network. Each panel shows hallucinated spatial patterns that excite a specific neuron. (b) Closest images in the MODIS imagery dataset to each of the samples in (a), determined by cosine similarity in the image embedding space.

Citation: Artificial Intelligence for the Earth Systems 3, 1; 10.1175/AIES-D-23-0036.1

• Download Figure
• Download figure as PowerPoint slide

Activation maximization typically does not yield crisp or realistic images, instead producing images that repeat specific patterns learned by the CNN. As an example, performing activation maximization on an output neuron of a CNN trained to detect dogs, might produce outputs that are a mishmash of dog eyes, ears, and noses, but not anything that closely resembles a dog. More generally, we expect the process to show patterns that give clear indications of what excites a neuron, but perhaps look nothing like any samples in the training data (Simonyan and Zisserman 2014; Nguyen et al. 2019). In Fig. 3b we show a collection of images from the MODIS training dataset that have the closest vector embeddings [Eq. (2)] to the corresponding panels in Fig. 3a. There are clear similarities between the activation maximization patterns and corresponding cloud images. The upper-left panel in Fig. 3a for instance, has hallucinated a series of small bright objects overlaid with dimmer fuzzy structures, and this appears to correspond with cirrus overriding fair weather cumulus, a common occurrence in the trades. This experiment, while not particularly quantitative, provides an important confirmation that the CNN detects meaningful cloud structures. In the face of limited training data, transfer learning—fine-tuning a CNN trained on a different dataset to operate on a new one—is a common and effective practice (Goodfellow et al. 2016). While often successful (Yosinski et al. 2014; Yuan et al. 2020), the concept of classifying cloud imagery based on its resemblance to ImageNet classes (e.g., vehicles, plants, animals) seems dubious, particularly in a scientific setting. This experiment demonstrates that the CNN has learned feature representations relevant to classifying cloud morphology. The CNNs trained in this work are deep neural networks and inherit the same interpretation difficulties as all deep CNNs. Nonetheless, the fact that their learned internal representations are clearly representative of different cloud types is likely desirable from an artificial intelligence trustworthiness and interpretability standpoint (McGovern et al. 2022; Ebert-Uphoff and Hilburn 2020) and indicates that self-supervised methods are a promising way to pretrain models for domain specific transfer learning.

d. Sensitivity to scene brightness

As a consequence of the aggressive data augmentation used to train the model, we expect that the CNN’s vector embeddings are robust against changing image brightness, and in the context of cloud imagery, this is a useful property for several downstream tasks. Many conventional passive satellite cloud retrieval algorithms are extremely sensitive to changes in observed spectral radiance. For instruments that are in orbit for long periods of time, even mild sensor degradation can lead to spurious trends in cloud properties derived from that sensor. As an example, Lyapustin et al. (2014) describe how a small long-term calibration degradation in Terra MODIS’s visible and near-infrared bands affected dependent retrievals, including a −13% per decade false trend in marine cloud optical thickness, a key parameter used to assess cloud type and radiative effects. Furthermore, interinstrument differences in calibration and hardware can be a major hurdle for generating multi-instrument records. Finally, for satellite imagers, scene illumination can change drastically depending on solar zenith angle and viewing angle. Therefore, classification invariance under different scene illumination or observed spectral radiance is a useful feature.

Here, we evaluate the stability of the vector embeddings generated by the self-supervised CNN under extreme changes in image brightness. First, the cosine similarity is computed between the vector embedding of each image in each labeled MODIS imagery dataset and the nearest-neighbor vector embedding from the same dataset. Then, copies of each dataset with different brightness alterations ranging from −50% to 50% are generated, and the cosine similarity between each image embedding and the embedding of its brightness-altered copy is computed. The blue regions in Fig. 4 show the resulting cosine similarity as a function of image brightness changes, while the red regions show the nearest-neighbor cosine similarity for comparison. The two levels of shading encompass 90% and 50% of the samples in each labeled imagery dataset. Even for extreme changes in scene brightness, the image vector embeddings remain relatively unaltered such that images remain closer to their brightness-altered counterparts in the embedding space than the nearest unaltered image from the same MODIS imagery dataset. While interpretation in terms of cosine similarity may be less intuitive than classification accuracy for instance, it demonstrates that the entire range of potential downstream uses for the vector embeddings (classification, clustering, searching, etc.) should be resilient to these kinds of changes in image properties.

View Full Size

Sensitivity of image vector embeddings to changing scene brightness. Blue represents the cosine similarity (vertical axis) between image vector embeddings and the embeddings of the same images with scaled brightness (horizontal axis). Red represents the cosine similarity between images and their nearest neighbor. The two levels of shading contain 90% and 50% of the samples from the testing datasets and the thick lines represent the mean. Higher values indicate greater similarity. The plot indicates that even for substantial changes in scene brightness, image vector embeddings remain relatively stable, and their classification would likely remain unchanged for most classification schemes.

Citation: Artificial Intelligence for the Earth Systems 3, 1; 10.1175/AIES-D-23-0036.1

• Download Figure
• Download figure as PowerPoint slide