Deep Learning for the Earth Sciences

Gustau Camps-Valls is Professor of Electrical Engineering and Lead Researcher in the Image Processing Laboratory (IPL) at the Universitat de València. His interests include development of statistical learning, mainly kernel machines and neural networks, for Earth sciences, from remote sensing to geoscience data analysis. Models efficiency and accuracy but also interpretability, consistency and causal discovery are driving his agenda on AI for Earth and climate.Devis Tuia, PhD, is Associate Professor at the Ecole Polytechnique Fédérale de Lausanne (EPFL). He leads the Environmental Computational Science and Earth Observation laboratory, which focuses on the processing of Earth observation data with computational methods to advance Environmental science. Xiao Xiang Zhu is Professor of Data Science in Earth Observation and Director of the Munich AI Future Lab AI4EO at the Technical University of Munich and heads the Department EO Data Science at the German Aerospace Center. Her lab develops innovative machine learning methods and big data analytics solutions to extract large scale geo-information from big Earth observation data, aiming at tackling societal grand challenges, e.g. Urbanization, UN’s SDGs and Climate Change. Markus Reichstein is Director of the Biogeochemical Integration Department at the Max-Planck- Institute for Biogeochemistry and Professor for Global Geoecology at the University of Jena. His main research interests include the response and feedback of ecosystems (vegetation and soils) to climatic variability with an Earth system perspective, considering coupled carbon, water and nutrient cycles. He has been tackling these topics with applied statistical learning for more than 15 years.

• Foreword xviby Vipin Kumar, Regents Professor, University of Minnesota Acknowledgments xviiList of Contributors xviiiList of Acronyms xxiv1 Introduction 1Gustau Camps-Valls, Xiao Xiang Zhu, Devis Tuia, and Markus Reichstein1.1 A Taxonomy of Deep Learning Approaches 21.2 Deep Learning in Remote Sensing 31.3 Deep Learning in Geosciences and Climate 71.4 Book Structure and Roadmap 9Part I Deep Learning to Extract Information from Remote Sensing Images 132 Learning Unsupervised Feature Representations of Remote Sensing Data with Sparse Convolutional Networks 15Jose E. Adsuara, Manuel Campos-Taberner, Javier García-Haro, Carlo Gatta, Adriana Romero, and Gustau Camps-Valls2.1 Introduction 152.2 Sparse Unsupervised Convolutional Networks 172.2.1 Sparsity as the Guiding Criterion 172.2.2 The EPLS Algorithm 182.2.3 Remarks 182.3 Applications 192.3.1 Hyperspectral Image Classification 192.3.2 Multisensor Image Fusion 212.4 Conclusions 223 Generative Adversarial Networks in the Geosciences 24Gonzalo Mateo-García, Valero Laparra, Christian Requena-Mesa, and Luis Gómez-Chova3.1 Introduction 243.2 Generative Adversarial Networks 253.2.1 Unsupervised GANs 253.2.2 Conditional GANs 263.2.3 Cycle-consistent GANs 273.3 GANs in Remote Sensing and Geosciences 283.3.1 GANs in Earth Observation 283.3.2 Conditional GANs in Earth Observation 303.3.3 CycleGANs in Earth Observation 303.4 Applications of GANs in Earth Observation 313.4.1 Domain Adaptation Across Satellites 313.4.2 Learning to Emulate Earth Systems from Observations 333.5 Conclusions and Perspectives 364 Deep Self-taught Learning in Remote Sensing 37Ribana Roscher4.1 Introduction 374.2 Sparse Representation 384.2.1 Dictionary Learning 394.2.2 Self-taught Learning 404.3 Deep Self-taught Learning 404.3.1 Application Example 434.3.2 Relation to Deep Neural Networks 444.4 Conclusion 455 Deep Learning-based Semantic Segmentation in Remote Sensing 46Devis Tuia, Diego Marcos, Konrad Schindler, and Bertrand Le Saux5.1 Introduction 465.2 Literature Review 475.3 Basics on Deep Semantic Segmentation: Computer Vision Models 495.3.1 Architectures for Image Data 495.3.2 Architectures for Point-clouds 525.4 Selected Examples 555.4.1 Encoding Invariances to Train Smaller Models: The example of Rotation 555.4.2 Processing 3D Point Clouds as a Bundle of Images: SnapNet 595.4.3 Lake Ice Detection from Earth and from Space 625.5 Concluding Remarks 666 Object Detection in Remote Sensing 67Jian Ding, Jinwang Wang, Wen Yang, and Gui-Song Xia6.1 Introduction 676.1.1 Problem Description 676.1.2 Problem Settings of Object Detection 696.1.3 Object Representation in Remote Sensing 696.1.4 Evaluation Metrics 696.1.4.1 Precision-Recall Curve 706.1.4.2 Average Precision and Mean Average Precision 716.1.5 Applications 716.2 Preliminaries on Object Detection with Deep Models 726.2.1 Two-stage Algorithms 726.2.1.1 R-CNNs 726.2.1.2 R-fcn 736.2.2 One-stage Algorithms 736.2.2.1 Yolo 736.2.2.2 Ssd 736.3 Object Detection in Optical RS Images 756.3.1 Related Works 756.3.1.1 Scale Variance 756.3.1.2 Orientation Variance 756.3.1.3 Oriented Object Detection 756.3.1.4 Detecting in Large-size Images 766.3.2 Datasets and Benchmark 776.3.2.1 Dota 776.3.2.2 VisDrone 776.3.2.3 Dior 776.3.2.4 xView 776.3.3 Two Representative Object Detectors in Optical RS Images 786.3.3.1 Mask OBB 786.3.3.2 RoI Transformer 826.4 Object Detection in SAR Images 866.4.1 Challenges of Detection in SAR Images 866.4.2 Related Works 866.4.3 Datasets and Benchmarks 886.5 Conclusion 897 Deep Domain Adaptation in Earth Observation 90Benjamin Kellenberger, Onur Tasar, Bharath Bhushan Damodaran, Nicolas Courty, and Devis Tuia7.1 Introduction 907.2 Families of Methodologies 917.3 Selected Examples 937.3.1 Adapting the Inner Representation 937.3.2 Adapting the Inputs Distribution 977.3.3 Using (few, well chosen) Labels from the Target Domain 1007.4 Concluding remarks 1048 Recurrent Neural Networks and the Temporal Component 105Marco Körner and Marc Rußwurm8.1 Recurrent Neural Networks 1068.1.1 Training RNNs 1078.1.1.1 Exploding and Vanishing Gradients 1078.1.1.2 Circumventing Exploding and Vanishing Gradients 1098.2 Gated Variants of RNNs 1118.2.1 Long Short-term Memory Networks 1118.2.1.1 The Cell State c t and the Hidden State h t 1128.2.1.2 The Forget Gate f t 1128.2.1.3 The Modulation Gate V T and the Input Gate I T 1128.2.1.4 The Output Gate o t 1128.2.1.5 Training LSTM Networks 1138.2.2 Other Gated Variants 1138.3 Representative Capabilities of Recurrent Networks 1148.3.1 Recurrent Neural Network Topologies 1148.3.2 Experiments 1158.4 Application in Earth Sciences 1178.5 Conclusion 1189 Deep Learning for Image Matching and Co-registration 120Maria Vakalopoulou, Stergios Christodoulidis, Mihir Sahasrabudhe, and Nikos Paragios9.1 Introduction 1209.2 Literature Review 1239.2.1 Classical Approaches 1239.2.2 Deep Learning Techniques for Image Matching 1249.2.3 Deep Learning Techniques for Image Registration 1259.3 Image Registration with Deep Learning 1269.3.1 2D Linear and Deformable Transformer 1269.3.2 Network Architectures 1279.3.3 Optimization Strategy 1289.3.4 Dataset and Implementation Details 1299.3.5 Experimental Results 1299.4 Conclusion and Future Research 1349.4.1 Challenges and Opportunities 1349.4.1.1 Dataset with Annotations 1349.4.1.2 Dimensionality of Data 1359.4.1.3 Multitemporal Datasets 1359.4.1.4 Robustness to Changed Areas 13510 Multisource Remote Sensing Image Fusion 136Wei He, Danfeng Hong, Giuseppe Scarpa, Tatsumi Uezato, and Naoto Yokoya10.1 Introduction 13610.2 Pansharpening 13710.2.1 Survey of Pansharpening Methods Employing Deep Learning 13710.2.2 Experimental Results 14010.2.2.1 Experimental Design 14010.2.2.2 Visual and Quantitative Comparison in Pansharpening 14010.3 Multiband Image Fusion 14310.3.1 Supervised Deep Learning-based Approaches 14310.3.2 Unsupervised Deep Learning-based Approaches 14510.3.3 Experimental Results 14610.3.3.1 Comparison Methods and Evaluation Measures 14610.3.3.2 Dataset and Experimental Setting 14610.3.3.3 Quantitative Comparison and Visual Results 14710.4 Conclusion and Outlook 14811 Deep Learning for Image Search and Retrieval in Large Remote Sensing Archives 150Gencer Sumbul, Jian Kang, and Begüm Demir11.1 Introduction 15011.2 Deep Learning for RS CBIR 15211.3 Scalable RS CBIR Based on Deep Hashing 15611.4 Discussion and Conclusion 159Acknowledgement 160Part II Making a Difference in the Geosciences with Deep Learning 16112 Deep Learning for Detecting Extreme Weather Patterns 163Mayur Mudigonda, Prabhat Ram, Karthik Kashinath, Evan Racah, Ankur Mahesh, Yunjie Liu, Christopher Beckham, Jim Biard, Thorsten Kurth, Sookyung Kim, Samira Kahou, Tegan Maharaj, Burlen Loring, Christopher Pal, Travis O’Brien, Kenneth E. Kunkel, Michael F. Wehner, and William D. Collins12.1 Scientific Motivation 16312.2 Tropical Cyclone and Atmospheric River Classification 16612.2.1 Methods 16612.2.2 Network Architecture 16712.2.3 Results 16912.3 Detection of Fronts 17012.3.1 Analytical Approach 17012.3.2 Dataset 17112.3.3 Results 17212.3.4 Limitations 17412.4 Semi-supervised Classification and Localization of Extreme Events 17512.4.1 Applications of Semi-supervised Learning in Climate Modeling 17512.4.1.1 Supervised Architecture 17612.4.1.2 Semi-supervised Architecture 17612.4.2 Results 17612.4.2.1 Frame-wise Reconstruction 17612.4.2.2 Results and Discussion 17812.5 Detecting Atmospheric Rivers and Tropical Cyclones Through Segmentation Methods 17912.5.1 Modeling Approach 17912.5.1.1 Segmentation Architecture 18012.5.1.2 Climate Dataset and Labels 18112.5.2 Architecture Innovations: Weighted Loss and Modified Network 18112.5.3 Results 18312.6 Challenges and Implications for the Future 18412.7 Conclusions 18513 Spatio-temporal Autoencoders in Weather and Climate Research 186Xavier-Andoni Tibau, Christian Reimers, Christian Requena-Mesa, and Jakob Runge13.1 Introduction 18613.2 Autoencoders 18713.2.1 A Brief History of Autoencoders 18813.2.2 Archetypes of Autoencoders 18913.2.3 Variational Autoencoders (VAE) 19113.2.4 Comparison Between Autoencoders and Classical Methods 19213.3 Applications 19313.3.1 Use of the Latent Space 19313.3.1.1 Reduction of Dimensionality for the Understanding of the System Dynamics and its Interactions 19513.3.1.2 Dimensionality Reduction for Feature Extraction and Prediction 19913.3.2 Use of the Decoder 19913.3.2.1 As a Random Sample Generator 20113.3.2.2 Anomaly Detection 20113.3.2.3 Use of a Denoising Autoencoder (DAE) Decoder 20213.4 Conclusions and Outlook 20314 Deep Learning to Improve Weather Predictions 204Peter D. Dueben, Peter Bauer, and Samantha Adams14.1 Numerical Weather Prediction 20414.2 How Will Machine Learning Enhance Weather Predictions? 20714.3 Machine Learning Across the Workflow of Weather Prediction 20814.4 Challenges for the Application of ML in Weather Forecasts 21314.5 The Way Forward 21615 Deep Learning and the Weather Forecasting Problem: Precipitation Nowcasting 218Zhihan Gao, Xingjian Shi, Hao Wang, Dit-Yan Yeung, Wang-chun Woo, and Wai-Kin Wong15.1 Introduction 21815.2 Formulation 22015.3 Learning Strategies 22115.4 Models 22315.4.1 FNN-based Odels 22315.4.2 RNN-based Models 22515.4.3 Encoder-forecaster Structure 22615.4.4 Convolutional LSTM 22615.4.5 ConvLSTM with Star-shaped Bridge 22715.4.6 Predictive RNN 22815.4.7 Memory in Memory Network 22915.4.8 Trajectory GRU 23115.5 Benchmark 23315.5.1 HKO-7 Dataset 23415.5.2 Evaluation Methodology 23415.5.3 Evaluated Algorithms 23515.5.4 Evaluation Results 23615.6 Discussion 236Appendix 238Acknowledgement 23916 Deep Learning for High-dimensional Parameter Retrieval 240David Malmgren-Hansen16.1 Introduction 24016.2 Deep Learning Parameter Retrieval Literature 24216.2.1 Land 24216.2.2 Ocean 24316.2.3 Cryosphere 24416.2.4 Global Weather Models 24416.3 The Challenge of High-dimensional Problems 24416.3.1 Computational Load of CNNs 24716.3.2 Mean Square Error or Cross-entropy Optimization? 24916.4 Applications and Examples 25016.4.1 Utilizing High-Dimensional Spatio-spectral Information with CNNs 25016.4.2 The Effect of Loss Functions in Retrieval of Sea Ice Concentrations 25316.5 Conclusion 25717 A Review of Deep Learning for Cryospheric Studies 258Lin Liu17.1 Introduction 25817.2 Deep-learning-based Remote Sensing Studies of the Cryosphere 26017.2.1 Glaciers 26017.2.2 Ice Sheet 26117.2.3 Snow 26217.2.4 Permafrost 26317.2.5 Sea Ice 26417.2.6 River Ice 26517.3 Deep-learning-based Modeling of the Cryosphere 26517.4 Summary and Prospect 266Appendix: List of Data and Codes 26718 Emulating Ecological Memory with Recurrent Neural Networks 269Basil Kraft, Simon Besnard, and Sujan Koirala18.1 Ecological Memory Effects: Concepts and Relevance 26918.2 Data-driven Approaches for Ecological memory Effects 27018.2.1 A Brief Overview of Memory Effects 27018.2.2 Data-driven Methods for Memory Effects 27118.3 Case Study: Emulating a Physical Model Using Recurrent Neural Networks 27218.3.1 Physical Model Simulation Data 27218.3.2 Experimental Design 27318.3.3 RNN Setup and Training 27418.4 Results and Discussion 27618.4.1 The Predictive Capability Across Scales 27618.4.2 Prediction of Seasonal Dynamics 27918.5 Conclusions 281Part III Linking Physics and Deep Learning Models 28319 Applications of Deep Learning in Hydrology 285Chaopeng Shen and Kathryn Lawson19.1 Introduction 28519.2 Deep Learning Applications in Hydrology 28619.2.1 Dynamical System Modeling 28619.2.1.1 Large-scale Hydrologic Modeling with Big Data 28619.2.1.2 Data-limited LSTM Applications 29019.2.2 Physics-constrained Hydrologic Machine Learning 29219.2.3 Information Retrieval for Hydrology 29319.2.4 Physically-informed Machine Learning for Subsurface Flow and Reactive Transport Modeling 29419.2.5 Additional Observations 29619.3 Current Limitations and Outlook 29620 Deep Learning of Unresolved Turbulent Ocean Processes in Climate Models 298Laure Zanna and Thomas Bolton20.1 Introduction 29820.2 The Parameterization Problem 29920.3 Deep Learning Parameterizations of Subgrid Ocean Processes 30020.3.1 Why DL for Subgrid Parameterizations? 30020.3.2 Recent Advances in DL for Subgrid Parameterizations 30020.4 Physics-aware Deep Learning 30120.5 Further Challenges ahead for Deep Learning Parameterizations 30321 Deep Learning for the Parametrization of Subgrid Processes in Climate Models 307Pierre Gentine, Veronika Eyring, and Tom Beucler21.1 Introduction 30721.2 Deep Neural Networks for Moist Convection (Deep Clouds) Parametrization 30921.3 Physical Constraints and Generalization 31221.4 Future Challenges 31422 Using Deep Learning to Correct Theoretically-derived Models 315PeterA.G.Watson22.1 Experiments with the Lorenz ’96 System 31722.1.1 The Lorenz’96 Equations and Coarse-scale Models 31822.1.1.1 Theoretically-derived Coarse-scale Model 31822.1.1.2 Models with ANNs 31922.1.2 Results 32022.1.2.1 Single-timestep Tendency Prediction Errors 32022.1.2.2 Forecast and Climate Prediction Skill 32122.1.3 Testing Seamless Prediction 32422.2 Discussion and Outlook 32422.2.1 Towards Earth System Modeling 32522.2.2 Application to Climate Change Studies 32622.3 Conclusion 32723 Outlook 328Markus Reichstein, Gustau Camps-Valls, Devis Tuia, and Xiao Xiang ZhuBibliography 331Index 401