Developing a Data-Driven Transfer Learning Model to Locate Tropical Cyclone Centers on Satellite Infrared Imagery
1. Introduction
Tropical cyclones (TCs) are intense weather processes generated over tropical oceans. After making landfall, TCs can cause mudslides, flash floods, and other disasters, which can cause great damage to people (Zhang and Li 2017; Fernandez et al. 2006). Therefore, the monitoring and forecasting of TCs is very important. In addition, TC intensity estimation, TC tracking, and TC forecasting require accurate TC center location (Olander and Velden 2007, 2019). Therefore, accurate TC center location is crucial for forecasters and emergency responders (Jaiswal and Kishtawal 2013; Hu et al. 2017; Lu et al. 2017).
Satellite remote sensing is widely used to locate TC centers because they have wide spatial and temporal coverage (Zheng et al. 2019). Based on the sensor type, existing TC center location methods are mainly classified as 1) infrared based (IR based) (Velden and Hawkins 2002), 2) synthetic aperture radar based (SAR based), and 3) microwave based (MIC based).
Existing IR-based and SAR-based methods include 1) the subjective empirical judgment method (Olander and Velden 2007; Dvorak 1975, 1984), 2) the threshold method (Fett and Brand 1975; Chaurasia et al. 2010; Jin et al. 2014; You et al. 2022), 3) the spiral curve method (Jaiswal and Kishtawal 2011; Lu et al. 2019; Shin et al. 2022), and 4) the cloud-derived wind method (Wood 1994; Zheng et al. 2019, 2016; Cecil and Biswas 2017; Hu et al. 2019; Liu and Zhang 2022; Y. Wang et al. 2021). Below, a concise overview of these methods is provided, with their limitations highlighted.
The Dvorak (Dvorak 1975) method is the most popular technique worldwide, which divides TCs into several classes according to their intensity, each corresponding to several special TC pattern maps. The forecaster obtains the location of the TC center concerning the pattern map. Although the method is widely used, it is highly subjective. Dvorak (Dvorak 1984) and Olander et al. (Olander and Velden 2007, 2019) have optimized and automated the Dvorak method.
The threshold method is based on work by Fett and Brand (1975) and Chaurasia et al. (2010) that noted the structure of the central dense overcast of a TC is approximately elliptical. After the TC intensity reaches a certain level, the central dense overcast appears circular. The location of the TC center can be judged by observing the morphological features of the central dense overcast. The threshold method segments and identifies the central dense overcast and defines the morphological center of the central dense overcast as the TC center. This method requires complex preprocessing of the image and does not apply to TCs without an obvious central dense overcast.
The spiral curve method makes use of the vortex structure of a TC, from which the TC center can be determined by extracting the spiral curve of the TC cloud system. This method requires thresholding, contrast enhancement, histogram equalization, Gaussian smoothing, and filtering satellite infrared images before matching the spiral curve and identifying the spiral center as the TC center (Jaiswal and Kishtawal 2011; Yurchak 2007). This method takes longer to calculate and only applies to TCs with distinct spiral cloud bands.
The cloud-derived wind method (Zheng et al. 2019, 2016; Hu et al. 2019; Liu and Zhang 2022; Y. Wang et al. 2021) extracts vector wind information from cloud positions retrieved from a time series of IR imagery. The wind field is then used to determine the TC center.
SAR imagery (Jin et al. 2014) and microwave data (Zhang et al. 2014; Hu et al. 2019) have been used for TC structural analysis work. However, SAR systems are carried on polar-orbiting satellites, which often can only observe a part of the TC. Additionally, the spatial resolution of microwave data is low, which poses challenges for TC center localization. IR instruments are found on geostationary satellites with a broad look angle and high repeat coverage, making this format more convenient for TC analysis and forecast work, despite a lower spatial resolution than SAR.
In addition to some of the challenges identified above, the IR methods also rely on computational processing that converts satellite images into high-level artificial features (e.g., texture features, thresholding, edge detection) and requires complex image preprocessing with long computation time. IR imagery can return large errors for locating TC centers when TCs are low intensity or have poorly defined structure. Therefore, developing accurate and efficient methods for TC center locations remains challenging.
A more recent computational processing development is deep learning (DL). It has powerful data mining capabilities (Li et al. 2020, 2022) and has been widely used for a variety of applications in remote sensing in recent years, such as TC intensity estimation (Chen et al. 2019; C. Wang et al. 2021; Zheng et al. 2022), TC wind radius estimation (Zhuo and Tan 2021; Wang et al. 2022), identification and forecasting of ocean phenomena, sea ice (Zhang and Li 2020; S. Zhang et al. 2022; X. Zhang et al. 2022), and other (Wang et al. 2022; Wu et al. 2023). Yang et al. (2019) applied DL techniques to study TC center locations. Wang et al. (2019) used the CNN-L model to locate the TC center from infrared satellite images and further improved the accuracy of the TC center location by combining the TC identification model and the TC center location model (Wang et al. 2020).
DL originated in computer vision, and with the increased computational power, DL models have evolved toward deeper and more complex directions (He et al. 2016). Along with increased depth and complexity, the accuracy of DL models utilized in various applications has also improved. However, it is important to note that DL models have many parameters, and the models can be easily overfitted if not supported by sufficient training data (Goodfellow et al. 2017).
Regarding the application of DL in TC research, for instance, for tasks like TC intensity or wind field estimation, it is noteworthy that the latest DL models like ResNet or GoogLeNet (Zhuo and Tan 2021) exhibit lower performance compared to older models like VGG-19 (Simonyan and Zisserman 2014). This result is primarily due to the challenge that more complex DL approaches face when dealing with limited training data.
To address this constraint in utilizing DL for TC analysis, one strategy is to employ a transfer learning (TL) approach. TL operates on the premise that the fundamental features (e.g., edges and textures) extracted by DL models for diverse tasks are shared or similar. DL models trained on extensive datasets possess more robust capabilities for extracting these fundamental features than those trained on smaller datasets. TL serves as an efficient remedy for the issue of limited training data by transferring the robust feature extraction capabilities from DL models trained on large datasets to other DL models. Consequently, TL is an effective solution for addressing the challenge of working with small training datasets in computational tasks. Li et al. (2019) proposed a visibility detection method based on TL. Jeon et al. (2020) achieved high-precision sea fog detection from GOCI images using TL. Han et al. (2022) improved the accuracy of radar-based rainfall nowcasting using TL. X. Zhang et al. (2022) used a TL approach for predicting internal wave amplitude that linked a laboratory dataset (888 samples) to a smaller observational dataset (121 samples) and improved prediction accuracy by 21% over the observational dataset alone. This result indicates that TL approaches are promising for oceanographic applications where observational data are often limited.
The ImageNet dataset (Yang et al. 2020) (https://image-net.org/about.php), which is commonly used in computer vision, has more than 14 million training samples. Complex models trained with the ImageNet dataset have powerful feature extraction capabilities. In many studies, the fundamental features may be used in other applications. For example, the contour and texture feature extraction capability learned from the ImageNet dataset can be used for TC contour and texture extraction. More accurate features such as contours and textures help locate the TC center. The approach proposed in this paper will improve locational accuracy of TC center identification by combining the greater effectiveness, albeit lower accuracy, of simpler DL models in a limited observational data context with enhanced feature recognition developed for other applications and integrated in using TL. The approach proposed in this paper is the first time the TL approach has been combined with DL for TC center identification.
To solve the small number of data point issues, we first obtain an ImageNet-based pretraining model. Then the model parameters of the convolutional layer of the pretraining model are transferred to the TC center location model. Finally, satellite infrared TC images are used for fine-tuning the TC center location model parameters. Thus, the reuse of the fundamental features is achieved, and the location accuracy of the TC center is greatly improved.
Our contributions can be summarized as follows:
-
Improved ResNet model adds the residual fully connected modules to better cope with the TC center location problems.
-
The proposed idea of TL across disciplinary domains transfers model knowledge from the computer vision domain to TC center location research, realizing the reuse of the fundamental feature extraction capability and greatly improving the model performance, and improving the accuracy of weak-intensity TC center location.
-
The proposed TL idea enables the small sample TC center location data to be applied to the large parameter DL model as well, and can better exploit the upper-performance limit of the large parameter DL model.
The dataset and the DL-based models are introduced in sections 2 and 3. After those, section 4 provides an analysis and discussion of the model results. Finally, section 5 presents the summaries.
2. Data and preprocessing
a. Geostationary satellite infrared imagery of TCs
The Himawari-8 (H-8) satellite, which was launched by the Japan Meteorological Agency (JMA) in October 2014, provided the TC satellite infrared imagery used in this investigation. Three visible (0.47–0.64 μm), 3 near-infrared (0.86–2.3 μm), and 10 thermal infrared (3.9–13.3 μm) bands were among the 16 bands of data provided by the Advanced Himawari Imager (AHI) on board H-8. With a temporal resolution of 10 min and a spatial resolution of 0.5–5 km, the H-8 imaging range covers the Pacific Ocean (Bessho et al. 2016). Lu et al. (2019) found that multichannel image fusion could improve the TC center location accuracy. Therefore, in this paper, we select channels 8 (6.2 μm), 13 (10.4 μm), and 15 (12.3 μm) with high transmittance near the atmospheric window for the TC center location study. As shown in Fig. 1, with the same color bar, different channels show different information. For example, compared to channels 13 and 15, channel 8 (water vapor channel) image has a lower brightness temperature in the areas with a large water vapor content. A total of 2450 images with 5 km spatial resolution were utilized to create a dataset of 97 TCs over the northwest Pacific Ocean from 2015 to 2018.
Brightness temperature (K) images from different channels: (a) channel 8, (b) channel 13, and (c) channel 15, with a spatial coverage of 1600 km × 1600 km.
Citation: Journal of Atmospheric and Oceanic Technology 40, 12; 10.1175/JTECH-D-23-0026.1
b. Best track dataset of TCs
The best track dataset for TCs provided by the China Meteorological Administration (CMA; https://tcdata.typhoon.org.cn/) was used to extract TC images with their corresponding labels (Ying et al. 2014). The location and intensity of TCs in the North Pacific are specified at 6-h intervals. After 2017, for TCs making landfall in China, the temporal resolution is improved to 3 h during the 24 h before its landfall.
c. Data preprocessing
After normalization, the TC images are randomly split 3:1:1 into training, validation, and test data (Table 1).
Number of training, validation, and test data for the DL-based TC center location model.
A 321 × 321 size image centered at the TC center locations provided by the CMA best track dataset for each TC image was extracted. Each training or validation image was reduced from 321 × 321 to 224 × 224 pixels, randomly shifted by 0–30 pixels up, down, left, and right three times (Fig. 2). The shift range was 0–30. Finally, we label the subimages with the number and orientation of the shifted pixels. For example, if an imaging center were shifted 5 pixels to the left and 10 pixels up, the image would be labeled as (−5, 10). It should be noted that the test image was also cropped only once. These manipulations resulted in an expanded dataset of 4410 training images, 1470 validation images, and 490 test images with a reduced image size of 224 × 224.
Random cut images: (a) original image (size: 321 × 321) and (b)–(d) randomly cropped images (size: 224 × 224).
Citation: Journal of Atmospheric and Oceanic Technology 40, 12; 10.1175/JTECH-D-23-0026.1
3. Deep learning model development
a. TC center location model configuration
Among DL models, the CNN model (LeCun et al. 2015) is good at capturing spatial correlation in images and can extract complex image features. CNN can accurately and quickly extract features such as TC contours, TC textures, and TC eyes. Therefore, CNN models are often used to extract TC information from satellite images. The CNN framework is often designed as a feed-forward network that updates the model weights by a back-propagation algorithm (LeCun et al. 2015). It is made up of a fully connected layer, a pooling layer, and a convolutional layer. The fully connected layer learns the intricate nonlinear relationship between features and outputs, while the convolutional layer extracts imagery features, and the pooling layer smooths these features using filtering techniques. As a result, CNN avoids complex image preprocessing and feature engineering (feature engineering refers to transforming, selecting, creating, or preparing data features) (LeCun et al. 2015). The Alexnet (Krizhevsky et al. 2017) was an improved model based on the CNN framework that halved the error rate of target recognition. After that, VGG-19 (Simonyan and Zisserman 2014) deepened the model depth using small convolutional kernels; ResNet (He et al. 2016) further refined this approach by introducing a residual module that significantly deepened the model complexity, allowing the model to generalize and handle more complex tasks. Therefore, the ResNet model is selected as the base model for the TC center location model developed in this paper.
The ResNet model was originally designed for image classification work in computer vision. The convolutional layer structure of the ResNet model is kept unchanged so that the model can inherit the feature extraction capability of the pretrained model using TL. Two modifications (ResNet-TCL-A and ResNet-TCL-B) are introduced here to make it more suitable for the TC center location (Fig. 3).
Architecture of ResNet-TCL, ResNet-TCL-A, and ResNet-TCL-B model for locating TC center (FC means the fully connected layer).
Citation: Journal of Atmospheric and Oceanic Technology 40, 12; 10.1175/JTECH-D-23-0026.1
These are expanded upon below. First, the attention mechanism originated from the study of human vision and has helped to improve model performance in many studies (C. Wang et al. 2021; Wang et al. 2022). Therefore, ResNet-TCL-A adds the spatial and temporal attention layers between the input layer and the first convolutional layer. Second, each node of the fully connected layer connects all nodes of the previous layer, giving the model have stronger learning ability. Compared with the commonly used DL models (i.e., VGGNet), the ResNet model is not designed with a fully connected layer. Therefore, ResNet-TCL-B adds 2 residual fully connected modules between the last convolutional layer and the output layer.
Other than model architecture, the performance of the DL model is positively correlated with the number of samples required for training. The amount of modeling data determines the lower limit of model performance (Pan and Yang 2010). The problem of an insufficient amount of training data has been observed in many fields. This issue spurred the development of TL techniques.
TL is an important tool to solve the problem of insufficient training data and consists of two concepts: source and target domains. The target domain is the knowledge that has to be learned, and the source domain is the knowledge that already exists. TL is the application of knowledge or patterns learned in the source domain to the target domain (Pan and Yang 2010). The requirements for the data needed for DL are relaxed by TL. TL enables the reuse of the fundamental features, greatly reducing the data and training time required for the target task (Pan and Yang 2010; Hu et al. 2018).
In summary, once the model architecture has been established, the next major step is to train the TC center location model using the domain-adversarial TL (Fig. 4). The training steps of the TC center location model (ResNet-TCL, ResNet-TCL-A, ResNet-TCL-B) proposed in this paper include five steps:
Modeling process of the ResNet model based on TL.
Citation: Journal of Atmospheric and Oceanic Technology 40, 12; 10.1175/JTECH-D-23-0026.1
First, the pretrained models (ResNet-pre) trained on the ImageNet dataset, which consists of more than 14 million training samples and is a well-known dataset containing natural images for various object recognition tasks (Yang et al. 2020), are obtained (https://image-net.org/about.php). ResNet-pre follows the standard ResNet-50 structure, which includes 16 residual blocks, 49 convolutional layers, and 1 fully connected layer. ReLU is the activation function for all layers except the fully connected one. It is constructed using the Python TensorFlow-Keras programming language. Pretrained models based on the ImageNet dataset are offered by Keras, which can be directly downloaded without the need for retraining on the ImageNet dataset (for details, see https://keras.io/api/applications/resnet/).
Second, the TC center location model is built to locate the TC center. It is important to note that the TC center location model is used for all TCs, and the only similarity with the pretrained model (model-pre) is in the convolutional layer shown in Fig. 4.
Third, the weights of the convolutional layer in model-pre are transferred to TC center location model, and the weights of the TC center location model fully connected layer are initialized randomly.
Fourth, the TC center location dataset is used to fine-train TC center location model. It includes fine-tuning the weights of the convolutional layer and training the weights of the fully connected layer. It means that TC center location model inherits the pretrain model’s powerful fundamental feature extraction capability and improves the natural image feature extraction toward TC feature extraction during the fine-tuning process.
Fifth, the TL-based TC center location models are obtained.
b. Setup of experiments
First, the performance of the three model architectures for the TC center location is compared in Table 2 to determine the optimal model architecture suitable for the TC centers location. Then, three sets of experiments (Table 2) are set up in this paper to compare the model performance with and without TL to explore the role of TL on the ResNet-TCL, ResNet-TCL-A, and ResNet-TCL-B model.
Mean TC center location error (MAE) of ResNet-TCL, ResNet-TCL-A, and ResNet-TCL-B models with and without TL.
Once the optimal model architecture (ResNet-TCL-B, Table 2) has been established, the results in sections 4b–4d were all based on ResNet-TCL-B. In section 4b, the training data were divided into four equal parts. Four experiments were set up without TL, sequentially increasing the amount of training data; meanwhile, another set of four experiments was set up using TL with the same training data. How the size of the training dataset and the utilization of TL impact the accuracy of determining the center location was evaluated.
Note that the hyperparameter selection for the ResNet-TCL, ResNet-TCL-A, and ResNet-TCL-B models mentioned above is as follows, aiming to achieve fast convergence and high performance: 1) ReLU activation functions are used for the convolutional and fully connected layers to expedite model convergence. 2) The output layer employs a sigmoid activation function, the most commonly used activation function in regression model output layers. 3) The loss function chosen is mean-squared error (MSE), which closely aligns with the calculation method for distance (x2 + y2). 4) The optimization function is based on the adaptive moment estimation (Adam) algorithm. 5) The initial learning rate for the optimization function is set to 0.0005, and if the validation group’s loss value does not improve for eight consecutive epochs, the learning rate is increased by a factor of 0.5. Specifically, the initial learning rate of 0.0005 was selected to provide stable convergence, and the learning rate schedule helps prevent overfitting by adjusting the learning rate during training. This strategy was found effective during our experimentation. 6) The early stop function is applied during model training. If the validation data loss value does not decrease for over 10 epochs, the model’s training is stopped.
In addition, DL is often seen as a black box that does not explain the mechanism of how it works. Therefore, DL model interpretability is a key concern for scholars. The differences between TL and no-TL models using interpretability tools is discussed in section 4d.
4. Results and discussion
a. Modeling based on TL or no-TL
Table 2 and Fig. 5 show that the MAE of the ResNet-TCL, ResNet-TCL-A, and ResNet-TCL-B models are 34.5, 34.5, and 34.1 km. The results of the ResNet-TCL and ResNet-TCL-A models are comparable. It indicates that the attention layer does not help in the TC center location. The performance of the ResNet-TCL-B model without TL is 1.2% higher than that of ResNet-TCL. This is because the ResNet-TCL-B model has additional residual fully connected modules compared to the ResNet-TCL model. The residual fully connected modules allow the model to simulate complex tasks better.
MAE of ResNet-TCL, ResNet-TCL-A, and ResNet-TCL-B model with and without TL.
Citation: Journal of Atmospheric and Oceanic Technology 40, 12; 10.1175/JTECH-D-23-0026.1
Figure 6 shows the loss curves of the ResNet-TCL model for the training and validation data during the training process. The solid and dashed lines are the variation of the training and validation data loss values with the number of training epochs, respectively. Red and blue represents the ResNet-TCL model without and with TL, respectively. As shown in Fig. 6, the ResNet-TCL model with TL converged after 20 training epochs, while the ResNet-TCL model without TL started to converge only after 40 training epochs. Moreover, the loss values of the ResNet-TCL model with TL for both training and validation data are consistently lower than those of the ResNet-TCL model without TL. The results show that TL can accelerate model convergence and improve model performance.
Loss curve during the training of the ResNet-TCL model with and without TL.
Citation: Journal of Atmospheric and Oceanic Technology 40, 12; 10.1175/JTECH-D-23-0026.1
ResNet-TCL, ResNet-TCL-A, and ResNet-TCL-B models improved by 13.6% (29.8 km), 13.6% (29.8 km), and 14.1% (29.3 km) after using TL. The ResNet-TCL-B model has the lowest MAE and the largest performance improvement compared to the model without TL. At the same time, the residual fully connected modules of the ResNet-TCL-B model greatly increase the model parameters. More trainable parameters mean that the model requires more training data. TL can effectively address model data requirements and enable it to realize its upper-performance limit, especially for the ResNet-TCL-B model, fully.
b. The effect of training data on the model
The effect that size of the training dataset has on TL model performance is examined in this section. The best performing ResNet-TCL-B model in section 4a is chosen for the TL-based model in this section. The validation data and test data are consistent with section 4a. First, the training data (not randomly cropped) were divided into four equal parts, and then each part was randomly cropped three times (as described in section 3c) to expand the data. As shown in Table 3, ResNet-TCL-1 to ResNet-TCL-1 were trained using 25%, 50%, 75%, and 100% of the data.
TC center location results are based on different amounts of training data.
Table 3 and Fig. 7 show the effect of the amount of training data on the TL and no-TL models. The ResNet-TCL-1 to ResNet-TCL-4 mean location errors are 56.1, 43.8, 37.0, and 33.9 km when TL is not used. The model performance is improved as the training data increase, at a decreasing rate. When the training data were increased from 25% to 50%, the model performance was improved by 21.9%. When the training data were increased from 75% to 100%, the model performance improved by only 8.4%. The results show that the model performance tends to increase logarithmically with the number of training data.
TC center location results based on different amounts of training data.
Citation: Journal of Atmospheric and Oceanic Technology 40, 12; 10.1175/JTECH-D-23-0026.1
The model performance of ResNet-TCL-1 to ResNet-TCL-4 with TL was improved by 15.8%, 15.3%, 14.6%, and 13.6% compared to that without TL. The results show that TL can effectively improve the model performance. In particular, the highest improvement is achieved when the number of training data is small, and model performance improvement only slightly decreases with the increase in training data. When the amount of training data is large, such as ResNet-TCL-3, adding 25% of data improves the model performance by 8.4%, and using TL improves the model performance by 14.6%. The results show that the improvement of model performance by strong fundamental feature extraction capability is greater than that by increasing the amount of data.
Figure 8 shows the location results of the ResNet-TCL-4 with/without the TL model. Weak-intensity TCs have inconspicuous structures that are difficult to locate. Strong TCs have obvious TC eyes and structure and are more easily located. As a result, the location error gradually decreases with the increase in TC intensity. The ResNet-TCL-4 model with TL has less MAE than the ResNet-TCL-4 model without TL at each intensity. For example, the MAE for H1–H5 is 20.0 km, with less than 20 km for H2–H5.
MAE of the ResNet-TCL-4 model with TL.
Citation: Journal of Atmospheric and Oceanic Technology 40, 12; 10.1175/JTECH-D-23-0026.1
c. Comparison with the latest technology
The performance of the ResNet-TCL-B model with TL in locating the TC center was compared with the latest methods listed in Table 4 in this section. The location accuracy of DL models is generally comparable to or surpasses that of these methods. However, the methods proposed by Zheng et al. (2019), Lu et al. (2019), Wang et al. (2020), Shin et al. (2022), and Liu and Zhang (2022) lack publicly available source code, making it impossible to compare them using the same dataset. Although the comparison in Table 4 is not equitable, the results show that our model has an overall good accuracy by the existing standard. On the other hand, the DL models from Wang et al. (2019) were retrained using the same dataset as this paper (Table 4). Compared with the DL model of Wang et al. (2019), the ResNet-TCL-B model with TL performs more accuracy. The VGG-19 model has also been tested with TL. The MAE of VGG-19 without/with TL is 34.8 and 31.4 km, respectively, which is higher than the ResNet model. The results show the effectiveness of the TL-based DL model proposed in this paper.
Comparison of the performances of our model with TL and other techniques in locating TC center from satellite infrared image.
d. Visualization and interpretation of the ResNet-TCL model
A DL model is often thought of as a black box. In this black box, DL models extract features from images and learn the laws from the features to the target task. However, DL/TL results ultimately must be related to physical processes. Therefore, the interpretability of DL models has become a popular research topic. However, the existing DL model for the TC center location study (Yang et al. 2019; Wang et al. 2019; Wang et al. 2020) neglects the interpretation of the model. Therefore, the ResNet-TCL-B model will be analyzed using DL interpretability methods. In this section, the aim is to 1) emphasize why TL-based deep learning models excel in TC center location and 2) identify potential error sources.
To our knowledge, the most popular and efficient techniques for understanding DL models are feature maps and heat maps. The feature map shows the features extracted from the input image by the convolutional layer. The input image impact on the model output can be seen in the heat map. Heat map techniques such as activation maximization analysis (Toms et al. 2020), network layer correlation propagation (Andersson et al. 2021), and sensitivity analysis (Espeholt et al. 2022) have been widely used in the field of geomatics to provide insight into the internal mechanisms of DL models. For example, Toms et al. (2020), based on the heat map, found that the most relevant region of the ENSO phase category identified by the DL model is consistent with the Niño-3.4 region. This result validates that the working mechanism of DL models is consistent and complementary with existing knowledge, offering the possibility of DL feeding atmospheric ocean science. The toolboxes developed by Kotikalapudi (Kotikalapudi et al. 2017) and Lundberg (Lundberg and Lee 2017) bring together a variety of model interpretable methods. Such toolboxes can be used to invoke interpretable methods to obtain feature maps and heat maps. It is important to note that most of the existing DL interpretable methods target classification problems, not regression problems. The saliency map method, which calculates the effect of pixel changes in the input image on the results, is used in this paper to obtain the heat map.
The ResNet-TCL-B model has 49 convolutional layers, and the deeper the convolutional layer, the more abstract the features extracted. Therefore, this section shows the feature maps extracted from the third and thirteenth convolutional layers. The third and thirteenth convolutional layers have 64 and 128 convolutional kernels, respectively, so 64 and 128 feature maps are extracted, respectively.
Figures 9a and 9b show the feature maps extracted from the third convolutional layer in the ResNet-TCL-B model; Figs. 9c and 9d show the feature maps extracted from the thirteenth convolutional layer in the ResNet-TCL-B model. The feature map shown in Fig. 9a blurs the difference between TC clouds and sea in the input image and does not extract important fundamental features for the TC center location. The feature maps in Fig. 9b show the texture and contour features of the TC. It is seen in Fig. 9b that the area close to the TC center and the TC contour features are extracted. Figure 9c shows the texture features and contour features of the TC. The feature maps in Fig. 9d become abstract, and the features of the TC eye can be seen in some of the feature maps.
Feature maps are generated from different layers in the ResNet-TCL-B model (the latitude–longitude scale of input TC case is 8.80°–24.80°N, 115.00°–131.00°E). (a) Third convolutional layer in ResNet-TCL-B model with TL. (b) Third convolutional layer in ResNet-TCL-B model without TL. (c) Thirteenth convolutional layer in ResNet-TCL-B model with TL. (d) Thirteenth convolutional layer in ResNet-TCL-B model without TL. The input data are three-channel (channels 8, 13, and 15) H-8 images (Fig. 1).
Citation: Journal of Atmospheric and Oceanic Technology 40, 12; 10.1175/JTECH-D-23-0026.1
The features extracted in Figs. 9b and 9c are more similar. It shows that the no-TL model cannot learn the fundamental features of the TC at the shallow convolutional layers and only learns the important fundamental features of TC (texture, contour, etc.) at the deeper layer. On the other hand, the TL model can learn the important fundamental features of the TC at a shallow convolutional layer so that more convolutional layers can be used to extract further and learn the relationship between the fundamental features and the TC center.
It is as if three steps are needed to solve a problem. The transferred learning model already has the empirical knowledge from step 1 to step 2 and only needs to go from step 2 to step 3. On the other hand, a no-TL model needs to go from step 1 to step 2 to step 3, and errors from step 1 to step 2 affect the result of step 3. TL enables the migration of the fundamental feature extraction capabilities, thus enabling faster and better problem-solving.
Figure 10 shows the location error and heat maps for different intensity images. The errors on the saliency heat maps in the middle and right columns of Fig. 10 are the location errors of the TL and no-TL models. The model location error is small for H1–H5 TC with obvious TC eyes. For the lower-intensity TS TC, the model location error is larger. It is mainly because low-intensity TCs do not have significant circulation characteristics. Therefore, most TC center location methods cannot accurately locate the center with low intensity. For operational activities, the wind field simulated by the numerical model is usually combined to assist in locating the TC center.
Original images and saliency heat maps of TS (15.25°–26.40°N, 125.70°–136.85°E), H1 (25.10°–36.25°N, 122.70°–133.85°E), H2 (11.65°–22.80°N, 117.85°–129.00°E), H3 (9.80°–20.95°N, 137.20°–148.35°E), H4 (10.90°–22.05°N, 131.45°–142.60°E), and H5 (15.20°–26.35°N, 118.20°–129.35°E) of the TL and no-TL models. (top to bottom) The samples of H5 to TS. (left) Input image, (center) saliency heat maps of no-TL model, and (right) saliency heat maps of TL model.
Citation: Journal of Atmospheric and Oceanic Technology 40, 12; 10.1175/JTECH-D-23-0026.1
In the following, the reasons for the poor location results of the DL model will be analyzed using saliency heat maps. In Fig. 10, the left column is the input image, and the red dots are the TC center locations. The center column is the heat map of the no-TL model. The right column is the heat map of the TL model. The values of the heat map range from 0 to 1. The larger the value, the greater the effect of the change in that point on the result.
In the H1–H5 images with obvious TC eyes, the region of attention of the TL model is concentrated near the TC eyes, while the region outside the TC eyes is almost 0. It indicates that the TL model learns the feature of TC eyes, which is the most important for the TC center location. However, the no-TL model focuses on more scattered regions, including image edges and TC peripheral cloud areas, in addition to the TC eye, which interferes with the TC center location. Especially for the H3 image, the no-TL model does not extract the features of the TC eye, which leads to a large location error. It is noted that for strong-intensity TCs (H1–H5) with clear TC eye, the ability to accurately capture the TC eye is considered a potential error source.
The DL model relies on recognizing the TC contour and morphology to locate the TC center in TS images where the TC eye is not obvious. The attention of the TL model is focused on the left side of the image where the TC eye is located, while the opposite is true for the no-TL model. The results show that for weaker-intensity TCs without distinct eye features, the extraction of features related to TC structure and spiral bands becomes a potential error source. The TL model has better feature extraction ability and feature learning ability and can accurately extract the most important features associated with the TC center location.
In summary, the interpretability analysis of DL models is essential. It can assist us in identifying potential error sources in TC center location and enhance our understanding of TCs. In the images with TC eyes, the TL model focuses on the region of TC eyes, which is consistent with the perception of the threshold method to locate the TC center. In images without TC eyes, ResNet-TCL-B can also locate the TC center more accurately by relying on TC morphological features, solving the drawback that traditional methods cannot be used for images without obvious TC eyes. The results show that the fundamental feature extraction capability from other fields can be transferred to the study of the TC center location, and the features related to the TC center can be extracted more accurately to achieve high-accuracy TC center location.
5. Conclusions
A limited availability of training data makes application of newer, large-parameter DL models to the TC location problem difficult. Instead, simpler DL models must be used, which imposes a minimum MAE on the accuracy of TC centers located with such methods. In this paper, a novel approach has been presented, which uses TL to transfer into the DL model information about structures and features in the image, which enhance rate of DL model convergence and improve location accuracy.
Adding the residual full connectivity modules in front of the output layer of the ResNet-TCL model improves the model performance. Compared with the ResNet-TCL model, the ResNet-TCL-B model can locate the TC center more accurately. The ResNet-TCL-B model improves the location accuracy by 1.2% over the ResNet-TCL model. The TL-based ResNet-TCL-B model improves the location accuracy by 14.1% (29.3 km) over the no-TL model. The performance of the ResNet-TCL model based on the domain knowledge of TC center location increases logarithmically with the amount of training data. When the number of samples is relatively small, increasing the number of training samples can greatly improve the model location accuracy. When the training samples are large, the benefit of increasing the training samples is smaller than the benefit of using TL. The visualization of the ResNet-TCL model shows that the TL model can accurately extract the most important features related to TC center location, including TC eye, TC texture, and contour.
The MAE of the ResNet-TCL-B model with TL for all TC cases in the test group is 29.3 km, 20 km for H1–H5 TCs, and less than 20 km for H2–H5 TCs. The location accuracy of the ResNet-TCL-B model with TL has improved by 15%–45% compared with the latest TC center location methods based on satellite infrared images. It can provide data support for TC monitoring.
The data available for TC research are much less than that for computer vision and other fields. Therefore, the cross-domain TL idea developed in this paper provides ideas for small-sample TC information extraction and modeling and can be used in other TC monitoring research, such as TC intensity estimation and TC wind radius estimation.
This paper uses the northwest Pacific Ocean for piloting and shows the potential of TL modeling. However, TCs in different seas have different characteristics, and a study examining a global TC dataset should be considered in the future. Further investigation into the interpretability of DL and its integration with TC research is crucial, and future studies in this field may focus on adding physical limitations or prior knowledge to DL models.
Acknowledgments.
This work was supported by the Qingdao National Laboratory for Marine Science and Technology, the special fund of Shandong Province (No. LSKJ202204302), Key Project of the Center for Ocean Mega-Science, Chinese Academy of Sciences (COMS2019R02), the Strategic Priority Research Program of the Chinese Academy of Sciences (XDB42000000), the National Natural Science Foundation of China (U2006211), and the Major scientific and technological innovation projects in Shandong Province (2019JZZY010102).
Data availability statement.
Himawari-8 geostationary satellite data are obtained from Japan Meteorological Agency (http://www.jma.go.jp/). The best track data of tropical cyclones (TCs) are from the Shanghai Typhoon Institute, China Meteorological Administration (http://www.cma.gov.cn/).
REFERENCES
Andersson, T. R., and Coauthors, 2021: Seasonal Arctic sea ice forecasting with probabilistic deep learning. Nat. Commun., 12, 5124, https://doi.org/10.1038/s41467-021-25257-4.
Bessho, K., and Coauthors, 2016: An introduction to Himawari-8/9—Japan’s new-generation geostationary meteorological satellites. J. Meteor. Soc. Japan, 94, 151–183, https://doi.org/10.2151/jmsj.2016-009.
Cecil, D. J., and S. K. Biswas, 2017: Hurricane Imaging Radiometer (HIRAD) wind speed retrievals and validation using dropsondes. J. Atmos. Oceanic Technol., 34, 1837–1851, https://doi.org/10.1175/JTECH-D-17-0031.1.
Chaurasia, S., C. M. Kishtawal, and P. K. Pal, 2010: An objective method of cyclone centre determination from geostationary satellite observations. Int. J. Remote Sens., 31, 2429–2440, https://doi.org/10.1080/01431160903012457.
Chen, B.-F., B. Chen, H.-T. Lin, and R. L. Elsberry, 2019: Estimating tropical cyclone intensity by satellite imagery utilizing convolutional neural networks. Wea. Forecasting, 34, 447–465, https://doi.org/10.1175/WAF-D-18-0136.1.
Dvorak, V. F., 1975: Tropical cyclone intensity analysis and forecasting from satellite imagery. Mon. Wea. Rev., 103, 420–430, https://doi.org/10.1175/1520-0493(1975)103<0420:TCIAAF>2.0.CO;2.
Dvorak, V. F., 1984: Tropical cyclone intensity analysis using satellite data. NOAA Tech. Rep. NESDIS 11, 47 pp., http://satepsanone.nesdis.noaa.gov/pub/Publications/Tropical/Dvorak_1984.pdf.
Espeholt, L., and Coauthors, 2022: Deep learning for twelve hour precipitation forecasts. Nat. Commun., 13, 5145, https://doi.org/10.1038/s41467-022-32483-x.
Fernandez, D. E., J. R. Carswell, S. Frasier, P. S. Chang, P. G. Black, and F. D. Marks, 2006: Dual-polarized C- and Ku-band ocean backscatter response to hurricane-force winds. J. Geophys. Res., 111, C08013, https://doi.org/10.1029/2005JC003048.
Fett, R. W., and S. Brand, 1975: Tropical cyclone movement forecasts based on observations from satellites. J. Appl. Meteor., 14, 452–465, https://doi.org/10.1175/1520-0450(1975)014%3C0452:TCMFBO%3E2.0.CO;2.
Goodfellow, I., Y. Bengio, and A. Courville, 2017: Deep Learning. Vol. 1. MIT Press, 800 pp.
Han, L., Y. Zhao, H. Chen, and V. Chandrasekar, 2022: Advancing radar nowcasting through deep transfer learning. IEEE Trans. Geosci. Remote Sens., 60, 4100609, https://doi.org/10.1109/TGRS.2021.3056470.
He, K., X. Zhang, S. Ren, and J. Sun, 2016: Deep residual learning for image recognition. 2016 IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, NV, IEEE, 770–778, https://doi.org/10.1109/CVPR.2016.90.
Hu, J., L. Shen, and G. Sun, 2018: Squeeze-and-excitation networks. 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition, Salt Lake City, UT, IEEE, 7132–7141, https://doi.org/10.1109/CVPR.2018.00745.
Hu, T., X. Wang, D. Zhang, G. Zheng, Y. Zhang, Y. Wu, and B. Xie, 2017: Study on typhoon center monitoring based on HY-2 and FY-2 data. IEEE Geosci. Remote Sens. Lett., 14, 2350–2354, https://doi.org/10.1109/LGRS.2017.2764620.
Hu, T., Y. Wu, G. Zheng, D. Zhang, Y. Zhang, and Y. Li, 2019: Tropical cyclone center automatic determination model based on HY-2 and QuikSCAT wind vector products. IEEE Trans. Geosci. Remote Sens., 57, 709–721, https://doi.org/10.1109/TGRS.2018.2859819.
Jaiswal, N., and C. M. Kishtawal, 2011: Automatic determination of center of tropical cyclone in satellite-generated IR images. IEEE Geosci. Remote Sens. Lett., 8, 460–463, https://doi.org/10.1109/LGRS.2010.2085418.
Jaiswal, N., and C. M. Kishtawal, 2013: Objective detection of center of tropical cyclone in remotely sensed infrared images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 6, 1031–1035, https://doi.org/10.1109/JSTARS.2012.2215016.
Jeon, H.-K., S. Kim, J. Edwin, and C.-S. Yang, 2020: Sea fog identification from GOCI images using CNN transfer learning models. Electronics, 9, 311, https://doi.org/10.3390/electronics9020311.
Jin, S., S. Wang, and X. Li, 2014: Typhoon eye extraction with an automatic SAR image segmentation method. Int. J. Remote Sens., 35, 3978–3993, https://doi.org/10.1080/01431161.2014.916447.
Kotikalapudi, R., and Coauthors, 2017: Keras-vis. GitHub, https://github.com/raghakot/keras-vis.
Krizhevsky, A., I. Sutskever, and G. E. Hinton, 2017: ImageNet classification with deep convolutional neural networks. Commun. ACM, 60, 84–90, https://doi.org/10.1145/3065386.
LeCun, Y., Y. Bengio, and G. Hinton, 2015: Deep learning. Nature, 521, 436–444, https://doi.org/10.1038/nature14539.
Li, Q., S. Tang, X. Peng, and Q. Ma, 2019: A method of visibility detection based on the transfer learning. J. Atmos. Oceanic Technol., 36, 1945–1956, https://doi.org/10.1175/JTECH-D-19-0025.1.
Li, X., and Coauthors, 2020: Deep-learning-based information mining from ocean remote-sensing imagery. Natl. Sci. Rev., 7, 1584–1605, https://doi.org/10.1093/nsr/nwaa047.
Li, X., Y. Zhou, and F. Wang, 2022: Advanced information mining from ocean remote sensing imagery with deep learning. J. Remote Sens., 2022, 9849645, https://doi.org/10.34133/2022/9849645.
Liu, J., and Q. Zhang, 2022: Objective detection of a tropical cyclone’s center using satellite image sequences in the northwest Pacific. Atmosphere, 13, 381, https://doi.org/10.3390/atmos13030381.
Lu, X., H. Yu, X. Yang, and X. Li, 2017: Estimating tropical cyclone size in the northwestern Pacific from geostationary satellite infrared images. Remote Sens., 9, 728, https://doi.org/10.3390/rs9070728.
Lu, X., H. Yu, X. Yang, X. Li, and J. Tang, 2019: A new technique for automatically locating the center of tropical cyclones with multi-band cloud imagery. Front. Earth Sci., 13, 836–847, https://doi.org/10.1007/s11707-019-0784-6.
Lundberg, S. M., and S.-I. Lee, 2017: A unified approach to interpreting model predictions. Proc. 31st Int. Conf. on Neural Information Processing Systems, Long Beach, CA, Association for Computing Machinery, https://dl.acm.org/doi/10.5555/3295222.3295230.
Olander, T. L., and C. S. Velden, 2007: The advanced Dvorak technique: Continued development of an objective scheme to estimate tropical cyclone intensity using geostationary infrared satellite imagery. Wea. Forecasting, 22, 287–298, https://doi.org/10.1175/WAF975.1.
Olander, T. L., and C. S. Velden, 2019: The advanced Dvorak technique (ADT) for estimating tropical cyclone intensity: Update and new capabilities. Wea. Forecasting, 34, 905–922, https://doi.org/10.1175/WAF-D-19-0007.1.
Pan, S. J., and Q. Yang, 2010: A survey on transfer learning. IEEE Trans. Knowl. Data Eng., 22, 1345–1359, https://doi.org/10.1109/TKDE.2009.191.
Shin, Y., J. Lee, J. Im, and S. Sim, 2022: An advanced operational approach for tropical cyclone center estimation using geostationary-satellite-based water vapor and infrared channels. Remote Sens., 14, 4800, https://doi.org/10.3390/rs14194800.
Simonyan, K., and A. Zisserman, 2014: Very deep convolutional networks for large-scale image recognition. arXiv, 1409.1556v6, https://doi.org/10.48550/arXiv.1409.1556.
Sola, J., and J. Sevilla, 1997: Importance of input data normalization for the application of neural networks to complex industrial problems. IEEE Trans. Nucl. Sci., 44, 1464–1468, https://doi.org/10.1109/23.589532.
Toms, B. A., E. A. Barnes, and I. Ebert-Uphoff, 2020: Physically interpretable neural networks for the geosciences: Applications to Earth system variability. J. Adv. Model. Earth Syst., 12, e2019MS002002, https://doi.org/10.1029/2019MS002002.
Velden, C., and J. Hawkins, 2002: The increasing role of weather satellites in tropical cyclone analysis and forecasting. Fifth Int. Workshop on Tropical Cyclones, Cairns, Australia, WMO.
Wang, C., Q. Xu, X. Li, G. Zheng, B. Liu, and Y. Cheng, 2019: An objective technique for typhoon monitoring with satellite infrared imagery. 2019 Photonics and Electromagnetics Research Symp.—Fall, Xiamen, China, IEEE, 3218–3221, https://doi.org/10.1109/PIERS-Fall48861.2019.9021497.
Wang, C., G. Zheng, X. Li, Q. Xu, B. Liu, and J. Zhang, 2021: Tropical cyclone intensity estimation from geostationary satellite imagery using deep convolutional neural networks. IEEE Trans. Geosci. Remote Sens., 60, 4101416, https://doi.org/10.1109/TGRS.2021.3066299.
Wang, G., X. Wang, X. Wu, K. Liu, Y. Qi, C. Sun, and H. Fu, 2022: A hybrid multivariate deep learning network for multistep ahead sea level anomaly forecasting. J. Atmos. Oceanic Technol., 39, 285–301, https://doi.org/10.1175/JTECH-D-21-0043.1.
Wang, P., P. Wang, C. Wang, Y. Yuan, and D. Wang, 2020: A center location algorithm for tropical cyclone in satellite infrared images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 13, 2161–2172, https://doi.org/10.1109/JSTARS.2020.2995158.
Wang, Y., G. Zheng, X. Li, L. Zhou, B. Liu, P. Chen, L. Ren, and X. Li, 2021: An automatic algorithm for estimating tropical cyclone centers in synthetic aperture radar imagery. IEEE Trans. Geosci. Remote Sens., 60, 4203716, https://doi.org/10.1109/TGRS.2021.3105705.
Wood, V. T., 1994: A technique for detecting a tropical cyclone center using a Doppler radar. J. Atmos. Oceanic Technol., 11, 1207–1216, https://doi.org/10.1175/1520-0426(1994)011<1207:ATFDAT>2.0.CO;2.
Wu, X., G. Han, W. Li, Q. Shao, and L. Cao, 2023: Deep learning–based prediction of Kuroshio path south of Japan. J. Atmos. Oceanic Technol., 40, 175–190, https://doi.org/10.1175/JTECH-D-22-0043.1.
Yang, K., K. Qinami, F.-F. Li, J. Deng, and O. Russakovsky, 2020: Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the ImageNet hierarchy. Proc. 2020 Conf. on Fairness, Accountability, and Transparency, Barcelona, Spain, Association for Computing Machinery, 547–558, https://doi.org/10.1145/3351095.3375709.
Yang, X., Z. Zhan, and J. Shen, 2019: A deep learning based method for typhoon recognition and typhoon center location. 2019 IEEE Int. Geoscience and Remote Sensing Symp., Yokohama, Japan, IEEE, 9871–9874, https://doi.org/10.1109/IGARSS.2019.8899322.
Ying, M., W. Zhang, H. Yu, X. Lu, J. Feng, Y. Fan, Y. Zhu, and D. Chen, 2014: An overview of the China Meteorological Administration tropical cyclone database. J. Atmos. Oceanic Technol., 31, 287–301, https://doi.org/10.1175/JTECH-D-12-00119.1.
You, Q., Z. Li, C. Qian, and T. Wang, 2022: A tropical cyclone center location method based on satellite image. Comput. Intell. Neurosci., 2022, 3747619, https://doi.org/10.1155/2022/3747619.
Yurchak, B. S., 2007: Description of cloud-rain bands in a tropical cyclone by a hyperbolic-logarithmic spiral. Russ. Meteor. Hydrol., 32, 8–18, https://doi.org/10.3103/S1068373907010025.
Zhang, D., Y. Zhang, T. Hu, B. Xie, and J. Xu, 2014: A comparison of HY-2 and QuikSCAT vector wind products for tropical cyclone track and intensity development monitoring. IEEE Geosci. Remote Sens. Lett., 11, 1365–1369, https://doi.org/10.1109/LGRS.2013.2293496.
Zhang, J. A., and X. Li, 2017: Tropical cyclone multiscale wind features from spaceborne synthetic aperture radar. Hurricane Monitoring with Spaceborne Synthetic Aperture Radar, Springer, 25–39, https://doi.org/10.1007/978-981-10-2893-9_2.
Zhang, S., Q. Xu, H. Wang, Y. Kang, and X. Li, 2022: Automatic waterline extraction and topographic mapping of tidal flats from SAR images based on deep learning. Geophys. Res. Lett., 49, e2021GL096007, https://doi.org/10.1029/2021GL096007.
Zhang, X., and X. Li, 2020: Combination of satellite observations and machine learning method for internal wave forecast in the Sulu and Celebes Seas. IEEE Trans. Geosci. Remote Sens., 59, 2822–2832, https://doi.org/10.1109/TGRS.2020.3008067.
Zhang, X., H. Wang, S. Wang, Y. Liu, W. Yu, J. Wang, Q. Xu, and X. Li, 2022: Oceanic internal wave amplitude retrieval from satellite images based on a data-driven transfer learning model. Remote Sens. Environ., 272, 112940, https://doi.org/10.1016/j.rse.2022.112940.
Zheng, G., J. Yang, A. K. Liu, X. Li, W. G. Pichel, and S. He, 2016: Comparison of typhoon centers from SAR and IR images and those from best track data sets. IEEE Trans. Geosci. Remote Sens., 54, 1000–1012, https://doi.org/10.1109/TGRS.2015.2472282.
Zheng, G., J. Liu, J. Yang, and X. Li, 2019: Automatically locate tropical cyclone centers using top cloud motion data derived from geostationary satellite images. IEEE Trans. Geosci. Remote Sens., 57, 10 175–10 190, https://doi.org/10.1109/TGRS.2019.2931795.
Zheng, Z., C. Hu, Z. Liu, J. Hao, Q. Hou, and X. Jiang, 2022: Deep learning for typhoon intensity classification using satellite cloud images. J. Atmos. Oceanic Technol., 39, 55–69, https://doi.org/10.1175/JTECH-D-19-0207.1.
Zhuo, J.-Y., and Z.-M. Tan, 2021: Physics-augmented deep learning to improve tropical cyclone intensity and size estimation from satellite imagery. Mon. Wea. Rev., 149, 2097–2113, https://doi.org/10.1175/MWR-D-20-0333.1.
