Evaluation, Tuning, and Interpretation of Neural Networks for Working with Images in Meteorological Applications

Imme Ebert-Uphoff Electrical and Computer Engineering, and Cooperative Institute for Research in the Atmosphere, Colorado State University, Fort Collins, Colorado

Search for other papers by Imme Ebert-Uphoff in
Current site
Google Scholar
PubMed
Close
and
Kyle Hilburn Cooperative Institute for Research in the Atmosphere, Colorado State University, Fort Collins, Colorado

Search for other papers by Kyle Hilburn in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

The method of neural networks (aka deep learning) has opened up many new opportunities to utilize remotely sensed images in meteorology. Common applications include image classification, e.g., to determine whether an image contains a tropical cyclone, and image-to-image translation, e.g., to emulate radar imagery for satellites that only have passive channels. However, there are yet many open questions regarding the use of neural networks for working with meteorological images, such as best practices for evaluation, tuning, and interpretation. This article highlights several strategies and practical considerations for neural network development that have not yet received much attention in the meteorological community, such as the concept of receptive fields, underutilized meteorological performance measures, and methods for neural network interpretation, such as synthetic experiments and layer-wise relevance propagation. We also consider the process of neural network interpretation as a whole, recognizing it as an iterative meteorologist-driven discovery process that builds on experimental design and hypothesis generation and testing. Finally, while most work on neural network interpretation in meteorology has so far focused on networks for image classification tasks, we expand the focus to also include networks for image-to-image translation.

© 2020 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Imme Ebert-Uphoff, iebert@colostate.edu

Abstract

The method of neural networks (aka deep learning) has opened up many new opportunities to utilize remotely sensed images in meteorology. Common applications include image classification, e.g., to determine whether an image contains a tropical cyclone, and image-to-image translation, e.g., to emulate radar imagery for satellites that only have passive channels. However, there are yet many open questions regarding the use of neural networks for working with meteorological images, such as best practices for evaluation, tuning, and interpretation. This article highlights several strategies and practical considerations for neural network development that have not yet received much attention in the meteorological community, such as the concept of receptive fields, underutilized meteorological performance measures, and methods for neural network interpretation, such as synthetic experiments and layer-wise relevance propagation. We also consider the process of neural network interpretation as a whole, recognizing it as an iterative meteorologist-driven discovery process that builds on experimental design and hypothesis generation and testing. Finally, while most work on neural network interpretation in meteorology has so far focused on networks for image classification tasks, we expand the focus to also include networks for image-to-image translation.

© 2020 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Imme Ebert-Uphoff, iebert@colostate.edu
Save
  • Alber, M., and Coauthors, 2019: iNNvestigate neural networks! J. Mach. Learn. Res., 20, 93, www.jmlr.org/papers/volume20/18-540/18-540.pdf.

    • Search Google Scholar
    • Export Citation
  • Alberga, V., 2009: Similarity measures of remotely sensed multi-sensor images for change detection applications. Remote Sens ., 1, 122143, https://doi.org/10.3390/rs1030122.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Araujo, A., W. Norris, and J. Sim, 2019: Computing receptive fields of convolutional neural networks. Distill, 4, e21, https://doi.org/10.23915/distill.00021.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Barnes, E. A., J. W. Hurrell, I. Ebert-Uphoff, C. Anderson, and D. Anderson, 2019: Viewing forced climate patterns through an AI lens. Geophys. Res. Lett., 46, 13 38913 398, https://doi.org/10.1029/2019GL084944.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Barnes, E. A., B. Toms, J. W. Hurrell, I. Ebert-Uphoff, C. Anderson, and D. Anderson, 2020: Indicator patterns of forced change learned by an artificial neural network. J. Adv. Model. Earth Syst., 12, e2019MS002002, https://doi.org/10.1029/2020MS002195.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Beucler, T., M. Pritchard, P. Gentine, and S. Rasp, 2020: Towards physically-consistent, data-driven models of convection. arXiv, https://arxiv.org/abs/2002.08525.

    • Search Google Scholar
    • Export Citation
  • Bonfanti, C., L. Trailovic, J. Stewart, and M. Govett, 2018: Machine learning: Defining worldwide cyclone labels for training. 21st Int. Conf. on Information Fusion, Cambridge, United Kingdom, IEEE, 753760, https://doi.org/10.23919/ICIF.2018.8455276.

    • Search Google Scholar
    • Export Citation
  • Boukabara, S.-A., V. Krasnopolsky, J. Q. Stewart, E. S. Maddy, N. Shahroudi, and R. N. Hoffman, 2019: Leveraging modern artificial intelligence for remote sensing and NWP: Benefits and challenges. Bull. Amer. Meteor. Soc., 100, ES473ES491, https://doi.org/10.1175/BAMS-D-18-0324.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Burkov, A., 2019: The Hundred-Page Machine Learning Book. Burkov, 160 pp.

  • Chollet, F., 2017: Deep Learning with Python. Manning Publications, 384 pp.

  • Cichy, R. M., and D. Kaiser, 2019: Deep neural networks as scientific models. Trends Cognit. Sci., 23, 305317, https://doi.org/10.1016/j.tics.2019.01.009.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cybenko, G., 1989: Approximation by superpositions of a sigmoidal function. Math. Contr. Signals Syst., 2, 303314, https://doi.org/10.1007/BF02551274.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Denby, L., 2020: Discovering the importance of mesoscale cloud organization through unsupervised classification. Geophys. Res. Lett., 47, e2019GL085190, https://doi.org/10.1029/2019GL085190.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gagne, D. J., II, A. McGovern, J. Brotzge, M. Coniglio, J. Correia Jr., and M. Xue, 2015: Day-ahead hail prediction integrating machine learning with storm-scale numerical weather models. 27th Conf. on Innovative Applications of Artificial Intelligence, Austin, TX, AAAI, 3954–3960.

    • Search Google Scholar
    • Export Citation
  • Gagne, D. J., S. E. Haupt, D. W. Nychka, and G. Thompson, 2019: Interpretable deep learning for spatial analysis of severe hailstorms. Mon. Wea. Rev., 147, 28272845, https://doi.org/10.1175/MWR-D-18-0316.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gagne, D. J., H. M. Christensen, A. C. Subramanian, and A. H. Monahan, 2020: Machine learning for stochastic parameterization: Generative adversarial networks in the Lorenz ’96 model. J. Adv. Model. Earth Syst., 12, e2019MS001896, https://doi.org/10.1029/2019MS001896.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Géron, A., 2019: Hands-on Machine Learning with Scikit-Learn, Keras, and Tensorflow: Concepts, Tools, and Techniques To Build Intelligent Systems. O’Reilly Media, 856 pp.

    • Search Google Scholar
    • Export Citation
  • Gonzalez, R. C., and R. E. Woods, 2002: Digital Image Processing. Prentice Hall, 793 pp.

  • Goodman, S. J., and Coauthors, 2013: The GOES-R Geostationary Lightning Mapper (GLM). Atmos. Res., 125–126, 3449, https://doi.org/10.1016/j.atmosres.2013.01.006.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hanin, B., 2019: Universal function approximation by deep neural nets with bounded width and ReLu activations. Mathematics, 7, 992, https://doi.org/10.3390/math7100992.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Haynes, J. M., C. Jakob, W. B. Rossow, G. Tselioudis, and J. Brown, 2011: Major characteristics of Southern Ocean cloud regimes and their effects on the energy budget. J. Climate, 24, 50615080, https://doi.org/10.1175/2011JCLI4052.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hertel, L., J. Collado, P. Sadowski, J. Ott, and P. Baldi, 2020: Sherpa: Robust hyperparameter optimization for machine learning. arXiv, https://arxiv.org/abs/2005.04048.

    • Crossref
    • Export Citation
  • Hilburn, K. A., I. Ebert-Uphoff, and S. D. Miller, 2020: Development and interpretation of a neural network-based synthetic radar reflectivity estimator using GOES-R satellite observations. J. Appl. Meteor. Climatol., 60, 321, https://doi.org/10.1175/JAMC-D-20-0084.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hornik, K., 1991: Approximation capabilities of multilayer feedforward networks. Neural Networks, 4, 251257, https://doi.org/10.1016/0893-6080(91)90009-T.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Isola, P., J.-Y. Zhu, T. Zhou, and A. A. Efros, 2017: Image-to-image translation with conditional adversarial networks. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Honolulu, HI, IEEE, 5967–5976, https://doi.org/10.1109/CVPR.2017.632.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Karpatne, A., W. Watkins, J. Read, and V. Kumar, 2017: Physics-guided neural networks (PGNN): An application in lake temperature modeling. arXiv, https://arxiv.org/abs/1710.11431.

  • Karpatne, A., I. Ebert-Uphoff, S. Ravela, H. A. Babaie, and V. Kumar, 2019: Machine learning for the geosciences: Challenges and opportunities. IEEE Trans. Knowl. Data Eng., 31, 15441554, https://doi.org/10.1109/TKDE.2018.2861006.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kasim, M., and Coauthors, 2020: Up to two billion times acceleration of scientific simulations with deep neural architecture search. arXiv, https://arxiv.org/abs/2001.08055.

  • Kim, K., J.-H. Kim, Y.-J. Moon, E. Park, G. Shin, T. Kim, Y. Kim, and S. Hong, 2019: Nighttime reflectance generation in the visible band of satellites. Remote Sens., 11, 2087, https://doi.org/10.3390/rs11182087.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kumler-Bonfanti, C., J. Stewart, D. Hall, and M. Govett, 2020: Tropical and extratropical cyclone detection using deep learning. arXiv, https://arxiv.org/abs/2005.09056.

    • Crossref
    • Export Citation
  • Lagerquist, R., A. McGovern, and D. J. Gagne II, 2019: Deep learning for spatially explicit prediction of synoptic-scale fronts. Wea. Forecasting, 34, 11371160, https://doi.org/10.1175/WAF-D-18-0183.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lagerquist, R., A. McGovern, C. R. Homeyer, D. J. Gagne, and T. Smith, 2020: Deep learning on three-dimensional multiscale data for next-hour tornado prediction. Mon. Wea. Rev., 148, 28372861, https://doi.org/10.1175/MWR-D-19-0372.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lapuschkin, S., S. Wäldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller, 2019: Unmasking clever HANS predictors and assessing what machines really learn. Nat. Commun., 10, 1096, https://doi.org/10.1038/s41467-019-08987-4.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lee, Y.-J., D. Hall, J. Stewart, and M. Govett, 2018: Machine learning for targeted assimilation of satellite data. Joint European Conf. on Machine Learning and Knowledge Discovery in Databases, Dublin, Ireland, ECML PKDD, 53–68.

    • Search Google Scholar
    • Export Citation
  • Lu, Z., H. Pu, F. Wang, Z. Hu, and L. Wang, 2017: The expressive power of neural networks: A view from the width. 31st Conf. on Neural Information Processing Systems, Long Beach, CA, NIPS, 6231–6239.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Luo, W., Y. Li, R. Urtasun, and R. Zemel, 2016: Understanding the effective receptive field in deep convolutional neural networks. 30th Conf. on Neural Information Processing Systems, Barcelona, Spain, NIPS, 48984906.

    • Search Google Scholar
    • Export Citation
  • McGovern, A., and R. A. Lagerquist, 2020: Using machine learning and model interpretation and visualization techniques to gain physical insights in atmospheric science. AI for Earth Sciences Workshop, ICLR, https://ai4earthscience.github.io/iclr-2020-workshop/papers/ai4earth16.pdf.

    • Search Google Scholar
    • Export Citation
  • McGovern, A., R. A. Lagerquist, D. J. Gagne, G. E. Jergensen, K. L. Elmore, C. R. Homeyer, and T. Smith, 2019: Making the black box more transparent: Understanding the physical implications of machine learning. Bull. Amer. Meteor. Soc., 100, 21752199, https://doi.org/10.1175/BAMS-D-18-0195.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Montavon, G., S. Lapuschkin, A. Binder, W. Samek, and K.-R. Müller, 2017: Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognit., 65, 211222, https://doi.org/10.1016/j.patcog.2016.11.008.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Montavon, G., W. Samek, and K.-R. Müller, 2018: Methods for interpreting and understanding deep neural networks. Digit. Signal Process., 73, 115, https://doi.org/10.1016/j.dsp.2017.10.011.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Olah, C., A. Mordvintsev, and L. Schubert, 2017: Feature visualization. Distill, 2, e7, https://doi.org/10.23915/distill.00007.

  • Olah, C., A. Satyanarayan, I. Johnson, S. Carter, L. Schubert, K. Ye, and A. Mordvintsev, 2018: The building blocks of interpretability. Distill, 3, e10, https://doi.org/10.23915/distill.00010.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Peng, D., Y. Zhang, and H. Guan, 2019: End-to-end change detection for high resolution satellite images using improved UNet++. Remote Sens ., 11, 1382, https://doi.org/10.3390/rs11111382.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Raghu, M., and E. Schmidt, 2020: A survey of deep learning for scientific discovery. arXiv, https://arxiv.org/abs/2003.11755.

  • Ramos-Pollán, R., N. de Posada, and M. A. G. López, 2011: Optimizing the area under the ROC curve in multilayer perceptron-based classifiers. Proc. Third Int. Conf. on Future Computational Technologies and Applications, Rome, Italy, IARIA, 7581.

    • Search Google Scholar
    • Export Citation
  • Reichstein, M., G. Camps-Valls, B. Stevens, M. Jung, J. Denzler, N. Carvalhais, and Prabhat, 2019: Deep learning and process understanding for data-driven Earth system science. Nature, 566, 195204, https://doi.org/10.1038/s41586-019-0912-1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Roebber, P. J., 2009: Visualizing multiple measures of forecast quality. Wea. Forecasting, 24, 601608, https://doi.org/10.1175/2008WAF2222159.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sahiner, B., X. He, W. Chen, H.-P. Chan, L. Hadjiiski, and N. Petrick, 2013: Neural network training by maximization of the area under the ROC curve: Application to characterization of masses on breast ultrasound as malignant or benign. Proc. SPIE, 8670, 86701M, https://doi.org/10.1117/12.2007615.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Samek, W., 2019: Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Vol. 11700. Springer Nature, 435 pp.

  • Schmit, T. J., P. Griffith, M. M. Gunshor, J. M. Daniels, S. J. Goodman, and W. J. Lebair, 2017: A closer look at the ABI on the GOES-R series. Bull. Amer. Meteor. Soc., 98, 681698, https://doi.org/10.1175/BAMS-D-15-00230.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Schrimpf, M., and Coauthors, 2018: Brain-score: Which artificial neural network for object recognition is most brain-like? bioRxiv, www.biorxiv.org/content/10.1101/407007v1.

    • Search Google Scholar
    • Export Citation
  • Selvaraju, R. R., M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, 2017: Grad-CAM: Visual explanations from deep networks via gradient-based localization. Proc. IEEE Int. Conf. on Computer Vision, Venice, Italy, IEEE, 618–626, https://doi.org/10.1109/ICCV.2017.74.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Simonyan, K., A. Vedaldi, and A. Zisserman, 2013: Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv, https://arxiv.org/abs/1312.6034.

  • Smilkov, D., N. Thorat, B. Kim, F. Viégas, and M. Wattenberg, 2017: SmoothGrad: Removing noise by adding noise. arXiv, https://arxiv.org/abs/1706.03825.

  • Smith, T. M., and Coauthors, 2016: Multi-Radar Multi-Sensor (MRMS) severe weather and aviation products: Initial operating capabilities. Bull. Amer. Meteor. Soc., 97, 16171630, https://doi.org/10.1175/BAMS-D-14-00173.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Snow, M., and Coauthors, 2016: Monge’s optimal transport distance for image classification. arXiv, https://arxiv.org/abs/1612.00181.

  • Sønderby, C. K., and Coauthors, 2020: MetNet: A neural weather model for precipitation forecasting. arXiv, https://arxiv.org/abs/2003.12140.

  • Stengel, K., A. Glaws, D. Hettinger, and R. N. King, 2020: Adversarial super-resolution of climatological wind and solar data. Proc. Natl. Acad. Sci. USA, 117, 16 80516 815, https://doi.org/10.1073/pnas.1918964117.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Toms, B. A., E. A. Barnes, and I. Ebert-Uphoff, 2020: Physically interpretable neural networks for the geosciences: Applications to Earth system variability. J. Adv. Model. Earth Syst., 12, e2019MS002002, https://doi.org/10.1029/2019MS002002.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tsagkatakis, G., A. Aidini, K. Fotiadou, M. Giannopoulos, A. Pentari, and P. Tsakalides, 2019: Survey of deep-learning approaches for remote sensing observation enhancement. Sensors, 19, 3929, https://doi.org/10.3390/s19183929.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, Z., A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, 2004: Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process ., 13, 600612, https://doi.org/10.1109/TIP.2003.819861.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Weyn, J. A., D. R. Durran, and R. Caruana, 2020: Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere. J. Adv. Model. Earth Syst., 12, e2020MS002109, https://doi.org/10.1029/2020MS002109.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Willard, J., X. Jia, S. Xu, M. Steinbach, and V. Kumar, 2020: Integrating physics-based modeling with machine learning: A survey. arXiv, https://arxiv.org/abs/2003.04919.

  • Wimmers, A., C. Velden, and J. H. Cossuth, 2019: Using deep learning to estimate tropical cyclone intensity from satellite passive microwave imagery. Mon. Wea. Rev., 147, 22612282, https://doi.org/10.1175/MWR-D-18-0391.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Xu, C., and B. Zhao, 2018: Satellite image spoofing: Creating remote sensing dataset with generative adversarial networks. 10th Int. Conf. on Geographic Information Science, Melbourne, Australia, Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 67, https://drops.dagstuhl.de/opus/volltexte/2018/9395/pdf/LIPIcs-GISCIENCE-2018-67.pdf.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 1 0 0
Full Text Views 20202 15452 4642
PDF Downloads 4501 536 47