Deep Reinforcement Learning and Its Industrial Use Cases

Shubham Mahajan, PhD, is an assistant professor in the School of Engineering at Ajeekya D Y Patil University, Pune, Maharashtra, India. He has eight Indian, one Australian, and one German patent to his credit in artificial intelligence and image processing. He has authored/co-authored more than 50 publications including peer-reviewed journals and conferences. His main research interests include image processing, video compression, image segmentation, fuzzy entropy, and nature-inspired computing methods with applications in optimization, data mining, machine learning, robotics, and optical communication. Pethuru Raj, PhD, is chief architect and vice president at Reliance Jio Platforms Ltd in Bangalore, India. He has a PhD in computer science and automation from the Indian Institute of Science in Bangalore, India. His areas of interest focus on artificial intelligence, model optimization, and reliability engineering. He has published thirty research papers and edited forty-two books. Amit Kant Pandit, PhD, is an associate professor in the School of Electronics & Communication Engineering Shri Mata Vaishno Devi University, India. He has authored/co-authored more than 60 publications including peer-reviewed journals and conferences. He has two Indian and one Australian patent to his credit in artificial intelligence and image processing. His main research interests are image processing, video compression, image segmentation, fuzzy entropy, and nature-inspired computing methods with applications in optimization.

• Preface xv1 Deep Reinforcement Learning Applications in Real-World Scenarios: Challenges and Opportunities 1Sunilkumar Ketineni and Sheela J.1.1 Introduction 11.1.1 Problems with Real-World Implementation 21.2 Application to the Real World 31.2.1 Security and Robustness 31.2.2 Generalization 51.2.2.1 Overcoming Challenges in DRL 91.3 Possibilities for Making a Difference in the Real World 111.3.1 Transfer Learning and Domain Adaptation 111.4 Meta-Learning 121.5 Deep Reinforcement Learning (DRL) 131.5.1 Hybrid Approaches 141.6 Online vs. Offline Reinforcement Learning 151.7 Human-in-the-Loop Systems 151.8 Benchmarking and Standardization 161.9 Collaborative Multi-Agent Systems 181.10 Transfer Learning and Domain Adaptation 191.11 Hierarchical and Multimodal Learning 211.12 Imitation Learning and Human Feedback 221.13 Inverse Reinforcement Learning 231.14 Sim-to-Real Transfer 241.15 Conclusion 25References 262 Deep Reinforcement Learning: A Key to Unlocking the Potential of Robotics and Autonomous Systems 29Saksham and Chhavi Rana2.1 Introduction 302.1.1 Significance of DRL Field 302.1.2 Transformative Advantages of DRL Field 322.2 Fields of Investigation 332.2.1 General Methods for Investigation 342.3 Background 362.3.1 Fundamentals of Deep Reinforcement Learning (DRL) 382.4 Deep Reinforcement Learning (DRL) in Robot Control 392.4.1 Navigation and Localization 402.4.2 Object Manipulation 422.5 Applications and Case Studies 432.6 Challenges and Future Directions 442.7 Evaluation and Metrics 462.8 Summary 47References 483 Deep Reinforcement Learning Algorithms: A Comprehensive Overview 51Shweta V. Bondre, Bhakti Thakre, Uma Yadav and Vipin D. Bondre3.1 Introduction 523.1.1 How Reinforcement Learning Works? 533.2 Reinforcement Learning Algorithms 533.2.1 Value-Based Algorithms 533.2.1.1 Q-Learning 533.2.1.2 Deep Q-Networks (DQN) 573.2.1.3 Double DQN 583.2.1.4 Dueling DQN 583.3 Policy-Based 593.3.1 Policy Gradient Methods 593.3.2 REINFORCE (Monte Carlo Policy Gradient) 603.3.3 Actor–Critic Methods 613.3.4 Natural Policy Gradient Methods 623.4 Model-Based Reinforcement Learning 633.4.1 Probabilistic Ensembles with Trajectory Sampling (PETS) 633.4.2 Probabilistic Inference for Learning Control (PILCO) 643.4.3 Model Predictive Control (MPC) 653.4.4 Model-Agnostic Meta-Learning (MAML) 663.4.5 Soft Actor–Critic with Model Ensemble 673.4.6 Deep Deterministic Policy Gradients with Model (DDPG with Model) 683.5 Characteristics of Reinforcement Learning 693.6 DRL Algorithms and Their Advantages and Drawbacks 713.7 Conclusion 72References 724 Deep Reinforcement Learning in Healthcare and Biomedical Applications 75Balakrishnan D., Aarthy C., Nandhagopal Subramani, Venkatesan R. and Logesh T. R.4.1 Introduction 764.2 Related Works 764.3 Deep Reinforcement Learning Framework 804.4 Deep Reinforcement Learning Applications in Healthcare and Biomedicine 814.5 Deep Reinforcement Learning Employs Efficient Algorithms 824.5.1 Deep Q-Networks 824.5.2 Policy Differentiation Techniques 824.5.3 Hindsight Experience Replay (HER) 824.5.4 Curiosity-Driven Exploration 824.5.5 Long Short-Term Memory Networks and Recurring Neural Network Designs 824.5.6 Multi-Agent DRL 834.6 Semi-Autonomous Control Based on Deep Reinforcement Learning for Robotic Surgery 834.6.1 Double Deep Q-Network (DDQN) 834.6.2 Materials and Methods 844.6.3 Results 864.6.4 Discussion 874.7 Conclusion 87References 885 Application of Deep Reinforcement Learning in Adversarial Malware Detection 91Manju and Chhavi Rana5.1 Introduction 915.1.1 Background 955.1.2 Significance of Malware Detection 965.1.3 Challenges with Adversarial Attacks 965.2 Foundations of Deep Reinforcement Learning 975.2.1 Overview of Deep Reinforcement Learning 985.2.2 Core Concepts and Components 995.2.3 Relevance to Malware Detection 1005.3 Malware Detection Landscape 1015.3.1 Evolution of Malware Detection Techniques 1025.3.2 Adversarial Attacks in Cybersecurity 1035.3.3 Need for Advanced Detection Strategies 1045.4 Deep Reinforcement Learning Techniques 1045.4.1 Application of Deep Learning in Malware Detection 1055.4.2 Reinforcement Learning Algorithms 1065.5 Feature Selection Strategies 1075.5.1 Importance of Feature Selection in Malware Detection 1085.5.2 Techniques for Feature Selection 1085.5.3 Optimization for Deep Reinforcement Learning Models 1095.6 Datasets and Evaluation 1105.7 Generating Adversarial Samples 111Conclusion and Future Directions 112Future Directions 112References 1126 Artificial Intelligence in Blockchain and Smart Contracts for Disruptive Innovation 115Eashwar Sivakumar, Kiran Jot Singh and Paras Chawla6.1 Introduction 1156.1.1 Smart Contract 1166.2 Literature Review 1176.2.1 Blockchain and Smart Contracts in Digital Identity 1176.2.2 Blockchain and Smart Contracts in Financial Security 1186.2.3 Blockchain and Smart Contracts in Supply Chain Management 1196.2.4 Blockchain and Smart Contracts in Insurance 1206.2.5 Blockchain and Smart Contracts in Healthcare 1216.2.6 Blockchain and Smart Contracts in Agriculture 1216.2.7 Blockchain and Smart Contracts in Real Estate 1226.2.8 Blockchain and Smart Contracts in Education and Research 1236.2.9 Blockchain and Smart Contracts in Other Sectors 1246.3 Critical Analysis of the Review 1256.4 Blockchain and Artificial Intelligence 1286.5 Discussion on the Reasoning for Implementation of Blockchain 1296.6 Conclusion 130References 1307 Clinical Intelligence: Deep Reinforcement Learning for Healthcare and Biomedical Advancements 137Keerthika K., Kannan M. and T. Saravanan7.1 Introduction 1387.2 Deep Reinforcement Learning Methods 1387.2.1 Model-Free Methods 1387.2.2 Policy Gradient Methods 1397.2.3 Model-Based Methods 1397.3 Applications of DRL in Healthcare 1407.3.1 Tailored Treatment Recommendations 1407.3.2 Optimization of Clinical Trials 1417.3.3 Disease Diagnosis Support 1427.3.4 Accelerated Drug Discovery and Design 1427.3.5 Enhanced Robotic Surgery and Assistance 1427.3.6 Health Management System 1437.4 Challenges 1437.5 Healthcare Data Types 1447.5.1 Electronic Healthcare Records (EHRs) 1447.5.2 Laboratory Data 1457.5.3 Sensor Data 1457.5.4 Biomedical Imaging Information 1457.6 Guidelines for the Application of DRL 1477.7 A Case Study: DRL in Healthcare and Biomedical Applications 1477.7.1 Optimizing Radiation Therapy Dose Distribution in Cancer Treatment 1477.7.2 Dose Strategy Model in Sepsis Patient Treatment 148References 1498 Cultivating Expertise in Deep and Reinforcement Learning Principles 151Chilakalapudi Malathi and J. Sheela8.1 Introduction 1518.1.1 Reinforcement Learning’s Constituent Parts 1528.1.2 Process of Markov Decisions (MDP) 1528.1.3 Learning Reinforcement Methods 1538.2 Intensive Learning Foundations 1648.2.1 A Definition of Deep Learning 1648.2.2 Deep Learning Elements 1648.2.2.1 Different Kinds of Deep Learning Networks 1658.3 Integrating Deep Learning and Reinforcement Learning 1728.3.1 Deep Reinforcement Learning 1728.3.2 Deep Reinforcement Learning Complexity Problems 174Conclusion 175References 1759 Deep Reinforcement Learning in Healthcare and Biomedical Research 179Shruti Agrawal and Pralay Mitra9.1 Introduction 1809.1.1 Reinforcement Learning 1809.1.2 Deep Reinforcement Learning 1819.2 Learning Methods in Bioinformatics with Applications in Healthcare and Biomedical Research 1829.2.1 Protein Folding 1829.2.2 Protein Docking 1839.2.3 Protein–Ligand Binding 1859.2.4 Binding Peptide Generation 1879.2.5 Protein Design and Engineering 1889.2.6 Drug Discovery and Development 1909.3 Applications in Biological Data 1929.3.1 Omics Data 1929.3.2 Medical Imaging 1929.3.3 Brain/Body–Machine Interfaces 1939.4 Adaptive Treatment Approach in Healthcare 1939.5 Diagnostic Tools in Healthcare and Biomedical Research 1959.6 Scope of Deep Reinforcement Learning in Healthcare and Biomedical Applications 1969.6.1 State and Action Space 1969.6.2 Reward 1979.6.3 Policy 1989.6.4 Model Training 1999.6.5 Exploration 1999.6.6 Credit Assignment 2009.7 Conclusions 200References 20110 Deep Reinforcement Learning in Robotics and Autonomous Systems 207Uma Yadav, Shweta V. Bondre and Bhakti Thakre10.1 Introduction 20810.2 The Promise of Deep Reinforcement Learning (DRL) in Real-World Robotics 21010.3 Preliminaries 21110.4 Enhancing RL for Real-World Robotics 22210.5 Reinforcement Learning for Various Robotic Applications 22410.6 Problems Faced in RL for Robotics 23110.7 RL in Robotics: Trends and Challenges 23210.8 Conclusion 235References 23611 Diabetic Retinopathy Detection and Classification Using Deep Reinforcement Learning 239H.R. Manjunatha and P. Sathish11.1 Introduction 23911.2 Literature Survey 24311.3 Diabetic Retinopathy Detection and Classification 24811.4 Result Analysis 25611.5 Conclusion 260References 26012 Early Brain Stroke Detection Based on Optimized Cuckoo Search Using LSTM‐Gated Multi-Perceptron Neural Network 265Anita Venaik, Asha A., Dhiyanesh B., Kiruthiga G., Shakkeera L. and Vinodkumar Jacob12.1 Introduction 26612.2 Literature Survey 26812.2.1 Problem Statement 26912.3 Proposed Methodology 27012.3.1 Dataset Collection 27012.3.2 Preprocessing 27112.3.3 Genetic Feature Sequence Algorithm (GFSA) 27512.3.4 Disease-Prone Factor (DPF) 28112.3.5 Decision Tree-Optimized Cuckoo Search (DTOCS) 28412.3.6 Long Short-Term Memory Gate Multilayer Perceptron Neural Network (LSTM-MLPNN) 28912.4 Result and Discussion 29312.4.1 Performance Matrix 29312.5 Conclusion 296References 29713 Hybrid Approaches: Combining Deep Reinforcement Learning with Other Techniques 301M. T. Vasumathi, Manju Sadasivan and Aurangjeb Khan13.1 Introduction 30213.1.1 Digital Twin—Introduction 30213.1.2 Model of a Digital Twin 30213.1.2.1 Steps Involved in Building a Digital Twin Prototype 30313.1.3 Application Areas of Digital Twins 30313.1.3.1 Digital Twin in Medical Field 30413.1.3.2 Digital Twin in Smart City 30413.1.3.3 Digital Twin in Sports 30413.1.3.4 Digital Twin in Smart Manufacturing 30513.2 Digital Twin Technologies 30513.2.1 Data Acquisition and Sensors 30613.2.2 Data Analytics and Machine Learning 30613.2.3 Cloud Computing 30713.2.4 Other Technologies 30713.3 Integration of RL and Digital Twin 30713.3.1 Motivation for Combining Digital Twin and RL 30913.3.2 How RL Enhances Decision-Making Within Digital Twins 31013.4 Challenges of Using RL in Digital Twins 31113.5 Digital Twin Modeling with RL 31213.6 Technology Underlying RL-Based Digital Twins 31413.6.1 Integration of RL with Digital Twins in Four Stages 31413.6.2 Tools and Libraries for Developing RL-Based Digital Twins 31413.6.2.1 Simulation and Digital Twin Platforms 31413.6.2.2 Reinforcement Learning Libraries 31513.6.3 Integration with Existing Systems and IoT Devices for RL Deployment 31513.6.3.1 Data Collection and Sensor Integration 31513.6.3.2 Communication and Data Ingestion 31613.6.3.3 Digital Twin Integration 31613.6.3.4 RL Integration 31613.6.3.5 Control and Actuation 31613.6.3.6 Implementation of Feedback and Learning Process 31613.6.3.7 Dashboard for Alert and Visualization 31613.6.3.8 Ensuring the Security and Authentication 31713.7 Industry-Specific Applications: A Case Study of DT in a Car Manufacturing Unit 31713.7.1 IoT Components Required for Creating Digital Twin for the Manufacturing Unit 31813.7.2 Architecture of the Proposed Digital Twin for Car Manufacturing Unit 31813.7.3 Challenges and Opportunities in the Implementation of DTs for Car Manufacturing 32013.8 Conclusion 321References 32214 Predictive Modeling of Rheumatoid Arthritis Symptoms: A High-Performance Approach Using HSFO-SVM and UNET-CNN 325Anusuya V., Baseera A., Dhiyanesh B., Parveen Begam Abdul Kareem and Shanmugaraja P.14.1 Introduction 32614.1.1 Novelty of the Research 32714.2 Related Work 32814.2.1 Challenges and Problem Identification Factor 33114.3 HSFO-SVM Based on LSTM-Gated Convolution Neural Network (lstmg-cnn) 33214.3.1 C-Score and Cross-Fold Validation 33214.3.2 Honey Scout Forager Optimization 33514.3.3 Feature Selection Using SVM 33614.3.4 UNET-CNN Classification 33814.4 Result and Discussion 34114.5 Conclusion 345References 34615 Using Reinforcement Learning in Unity Environments for Training AI Agent 349Geetika Munjal and Monika Lamba15.1 Introduction 34915.2 Literature Review 35115.3 Machine Learning 35215.3.1 Categorization of Machine Learning 35215.3.1.1 Supervised Learning 35215.3.1.2 Unsupervised Learning 35315.3.1.3 Reinforcement Learning 35315.3.2 Classifying on the Basis of Envisioned Output 35315.3.2.1 Classification 35415.3.2.2 Regression 35415.3.2.3 Clustering 35415.3.3 Artificial Intelligence 35415.4 Unity 35415.4.1 Unity Hub 35515.4.2 Unity Editor 35515.4.3 Inspector 35515.4.4 Game View 35515.4.5 Scene View 35515.4.6 Hierarchy 35515.4.7 Project Window 35615.5 Reinforcement Learning and Supervised Learning 35615.5.1 Positive Reinforcement 35715.5.2 Negative Reinforcement 35715.5.3 Model-Free and Model-Based RL 35715.6 Proposed Model 35915.6.1 Setting Up a Virtual Environment 35915.6.2 Setting Up of the Environment 36015.6.2.1 Creating and Allocating Scripts for the Environment 36115.6.2.2 Creating a Goal for the Agent 36115.6.2.3 Reward-Driven Behavior 36115.7 Markov Decision Process 36215.8 Model-Based RL 36215.9 Experimental Results 36315.9.1 Machine Learning Models Used for the Environments 36315.9.2 PushBlock 36315.9.3 Hallway 36515.9.4 Screenshots of the PushBlock Environment 36815.9.5 Screenshots of the Hallway Environment 36915.10 Conclusion 372References 37216 Emerging Technologies in Healthcare Systems 375Ravi Kumar Sachdeva, Priyanka Bathla, Samriti Vij, Dishika, Madhur Jain, Lokesh Kumar, G. S. Pradeep Ghantasala and Rakesh Ahuja16.1 Introduction 37516.2 Personalized Medicine 37616.3 AI and ML in Healthcare Sector 37716.3.1 AI in Medical Diagnosis 37816.3.2 Drug Discovery 37816.3.3 Personalized Treatment Plans 37916.3.4 Pattern Matching or Trend Detection 38016.4 Immunotherapy 38016.4.1 Monoclonal Antibodies 38116.4.2 Checkpoint Inhibitors 38116.4.3 CAR-T Cell Therapy 38116.5 Regenerative Medicine 38116.6 Digital Health (Use of Technology in Healthcare) 38316.6.1 Wearable Devices 38316.6.2 Telemedicine 38416.6.3 Electronic Health Records 38416.7 Health Inequity 38516.7.1 Health Disparity 38516.7.2 Health Equity 38516.8 Future Directions in Healthcare Research 38516.9 Challenges and Recommendations for Advanced Level of Modern Healthcare Technologies 38616.9.1 Challenges 38716.9.2 Recommendations 38816.10 Healthcare Sector in Developing and Underdeveloped Countries 38816.10.1 Healthcare Sector in Developing Countries 38816.10.2 Healthcare Sector in Underdeveloped Countries 38916.11 Comparison of Recent Progress and Future Mentoring in Healthcare Using Technology 38916.12 Conclusion 391References 392Index 395