Barouch Matzliach, PhD, is a Lecturer at Tel Aviv University, Faculty of Engineering, and a technology consultant to defense industries. He has thirty years of experience in the development and production of advanced land combat systems and is a recipient of the Israel Defense Award. His research at the LAMBDA laboratory focuses on Multi-Agent Reinforcement Learning and autonomous AI applications. Evgeny Kagan, PhD, CandSc-Eng, is a Senior Lecturer at the Department of Industrial Engineering, Ariel University, and a Research Fellow at LAMBDA laboratory. With over thirty years of experience in applied mathematics and engineering, he has authored more than eighty scientific publications including four books. Irad Ben-Gal, PhD, is a Full Professor at the Faculty of Engineering and a head of LAMBDA laboratory at Tel Aviv University and a world-renowned expert in data science, AI, and machine learning with over twenty-five years of academic and practical experience. He co-heads the TAU/Stanford University Digital Living 2030 initiative, has published four books, more than 150 scientific papers and patents, and has collaborated with Oracle, Intel, GM, AT&T, Applied Materials, Siemens, Kimberly Clark and Nokia.

• Preface ixAbout the Companion Website xi1 Introduction 11.1 The Problem of Search Under Uncertainty 11.2 Book Objectives and Main Contributions 21.3 Structure of the Book 32 Background 52.1 Probabilistic Search for Static and Moving Targets 62.2 Search in Shadowed Space 102.3 Search with False-Positive and False-Negative Errors 162.4 Conclusions 193 Problem Formulation and Basic Search Procedure 213.1 Basic Assumptions 213.2 Sensor's Fusion and Dynamic Probability Map 243.3 Search Policy and Sensing 263.3.1 Search Policy 263.3.2 Search Time and Sensors' Sensitivity 273.3.3 Numerical Simulations 283.4 Conclusions 304 Reactive Detection Algorithms 314.1 Agents' Policies and Decision-making 314.2 Policies Control and Brute-Force Learning 344.3 Numerical Simulations 354.3.1 Detection by a Single Agent 364.3.2 Detection by Multiple Agents 384.4 Conclusions 425 Search with Learning 435.1 Reinforcement Learning 435.2 Search with Supervised and Unsupervised Learning 475.3 Two-Actor Search 505.3.1 Pursuit-Evasion Game 515.3.2 Actor-Critic Approach 525.4 Conclusions 536 Search with Deep Q-Learning: Single Agent 556.1 Problem Formulation and Notation 556.2 Decision-making Policy and Deep Q-Learning Solution 586.2.1 Agent's Actions and Decision-making 586.2.2 Dynamic Programming and Target Neural Networks 596.2.3 Model-Free and Model-Based Learning 606.2.4 Choice of the Actions at the Learning Stage 636.2.5 The Q-max Algorithm 636.2.6 The SPL Algorithm 676.3 Numerical Simulations 676.3.1 Network Training in the Q-max Algorithm 686.3.2 Detection by the Q-max and SPL Algorithms 696.3.3 Comparison Between Q-max and SPL Algorithms and One-Step Heuristics 696.3.4 Comparison Between SPL Algorithm and Optimal Solution 736.3.5 Run Times and Mean-Squared Errors for Different Sizes of Data Sets 756.4 Conclusions 767 Search with Deep Q-learning: Multiple Agents 777.1 Problem Formulation and Notation 777.2 Cooperative Detection Using Voronoi Regions and Deep Q-Learning 787.2.1 Agents' Actions and Decision-making 797.2.2 Reactive Decision-making in Voronoi Regions: A Distributed EIG Algorithm 797.2.3 Collective Deep Q-Learning Approach 817.3 Numerical Simulations 887.3.1 Detection of Static Targets 897.3.2 Detection of Moving Targets 927.3.3 Learning Errors and Run Time of the Collective Q-max Algorithm 947.4 Conclusions 958 Conclusions 97References 99Index 103

Barouch Matzliach, PhD, is a Lecturer at Tel Aviv University, Faculty of Engineering, and a technology consultant to defense industries. He has thirty years of experience in the development and production of advanced land combat systems and is a recipient of the Israel Defense Award. His research at the LAMBDA laboratory focuses on Multi-Agent Reinforcement Learning and autonomous AI applications. Evgeny Kagan, PhD, CandSc-Eng, is a Senior Lecturer at the Department of Industrial Engineering, Ariel University, and a Research Fellow at LAMBDA laboratory. With over thirty years of experience in applied mathematics and engineering, he has authored more than eighty scientific publications including four books. Irad Ben-Gal, PhD, is a Full Professor at the Faculty of Engineering and a head of LAMBDA laboratory at Tel Aviv University and a world-renowned expert in data science, AI, and machine learning with over twenty-five years of academic and practical experience. He co-heads the TAU/Stanford University Digital Living 2030 initiative, has published four books, more than 150 scientific papers and patents, and has collaborated with Oracle, Intel, GM, AT&T, Applied Materials, Siemens, Kimberly Clark and Nokia.

• Preface ixAbout the Companion Website xi1 Introduction 11.1 The Problem of Search Under Uncertainty 11.2 Book Objectives and Main Contributions 21.3 Structure of the Book 32 Background 52.1 Probabilistic Search for Static and Moving Targets 62.2 Search in Shadowed Space 102.3 Search with False-Positive and False-Negative Errors 162.4 Conclusions 193 Problem Formulation and Basic Search Procedure 213.1 Basic Assumptions 213.2 Sensor's Fusion and Dynamic Probability Map 243.3 Search Policy and Sensing 263.3.1 Search Policy 263.3.2 Search Time and Sensors' Sensitivity 273.3.3 Numerical Simulations 283.4 Conclusions 304 Reactive Detection Algorithms 314.1 Agents' Policies and Decision-making 314.2 Policies Control and Brute-Force Learning 344.3 Numerical Simulations 354.3.1 Detection by a Single Agent 364.3.2 Detection by Multiple Agents 384.4 Conclusions 425 Search with Learning 435.1 Reinforcement Learning 435.2 Search with Supervised and Unsupervised Learning 475.3 Two-Actor Search 505.3.1 Pursuit-Evasion Game 515.3.2 Actor-Critic Approach 525.4 Conclusions 536 Search with Deep Q-Learning: Single Agent 556.1 Problem Formulation and Notation 556.2 Decision-making Policy and Deep Q-Learning Solution 586.2.1 Agent's Actions and Decision-making 586.2.2 Dynamic Programming and Target Neural Networks 596.2.3 Model-Free and Model-Based Learning 606.2.4 Choice of the Actions at the Learning Stage 636.2.5 The Q-max Algorithm 636.2.6 The SPL Algorithm 676.3 Numerical Simulations 676.3.1 Network Training in the Q-max Algorithm 686.3.2 Detection by the Q-max and SPL Algorithms 696.3.3 Comparison Between Q-max and SPL Algorithms and One-Step Heuristics 696.3.4 Comparison Between SPL Algorithm and Optimal Solution 736.3.5 Run Times and Mean-Squared Errors for Different Sizes of Data Sets 756.4 Conclusions 767 Search with Deep Q-learning: Multiple Agents 777.1 Problem Formulation and Notation 777.2 Cooperative Detection Using Voronoi Regions and Deep Q-Learning 787.2.1 Agents' Actions and Decision-making 797.2.2 Reactive Decision-making in Voronoi Regions: A Distributed EIG Algorithm 797.2.3 Collective Deep Q-Learning Approach 817.3 Numerical Simulations 887.3.1 Detection of Static Targets 897.3.2 Detection of Moving Targets 927.3.3 Learning Errors and Run Time of the Collective Q-max Algorithm 947.4 Conclusions 958 Conclusions 97References 99Index 103

Barouch Matzliach, PhD, is a Lecturer at Tel Aviv University, Faculty of Engineering, and a technology consultant to defense industries. He has thirty years of experience in the development and production of advanced land combat systems and is a recipient of the Israel Defense Award. His research at the LAMBDA laboratory focuses on Multi-Agent Reinforcement Learning and autonomous AI applications. Evgeny Kagan, PhD, CandSc-Eng, is a Senior Lecturer at the Department of Industrial Engineering, Ariel University, and a Research Fellow at LAMBDA laboratory. With over thirty years of experience in applied mathematics and engineering, he has authored more than eighty scientific publications including four books. Irad Ben-Gal, PhD, is a Full Professor at the Faculty of Engineering and a head of LAMBDA laboratory at Tel Aviv University and a world-renowned expert in data science, AI, and machine learning with over twenty-five years of academic and practical experience. He co-heads the TAU/Stanford University Digital Living 2030 initiative, has published four books, more than 150 scientific papers and patents, and has collaborated with Oracle, Intel, GM, AT&T, Applied Materials, Siemens, Kimberly Clark and Nokia.

• Preface ixAbout the Companion Website xi1 Introduction 11.1 The Problem of Search Under Uncertainty 11.2 Book Objectives and Main Contributions 21.3 Structure of the Book 32 Background 52.1 Probabilistic Search for Static and Moving Targets 62.2 Search in Shadowed Space 102.3 Search with False-Positive and False-Negative Errors 162.4 Conclusions 193 Problem Formulation and Basic Search Procedure 213.1 Basic Assumptions 213.2 Sensor's Fusion and Dynamic Probability Map 243.3 Search Policy and Sensing 263.3.1 Search Policy 263.3.2 Search Time and Sensors' Sensitivity 273.3.3 Numerical Simulations 283.4 Conclusions 304 Reactive Detection Algorithms 314.1 Agents' Policies and Decision-making 314.2 Policies Control and Brute-Force Learning 344.3 Numerical Simulations 354.3.1 Detection by a Single Agent 364.3.2 Detection by Multiple Agents 384.4 Conclusions 425 Search with Learning 435.1 Reinforcement Learning 435.2 Search with Supervised and Unsupervised Learning 475.3 Two-Actor Search 505.3.1 Pursuit-Evasion Game 515.3.2 Actor-Critic Approach 525.4 Conclusions 536 Search with Deep Q-Learning: Single Agent 556.1 Problem Formulation and Notation 556.2 Decision-making Policy and Deep Q-Learning Solution 586.2.1 Agent's Actions and Decision-making 586.2.2 Dynamic Programming and Target Neural Networks 596.2.3 Model-Free and Model-Based Learning 606.2.4 Choice of the Actions at the Learning Stage 636.2.5 The Q-max Algorithm 636.2.6 The SPL Algorithm 676.3 Numerical Simulations 676.3.1 Network Training in the Q-max Algorithm 686.3.2 Detection by the Q-max and SPL Algorithms 696.3.3 Comparison Between Q-max and SPL Algorithms and One-Step Heuristics 696.3.4 Comparison Between SPL Algorithm and Optimal Solution 736.3.5 Run Times and Mean-Squared Errors for Different Sizes of Data Sets 756.4 Conclusions 767 Search with Deep Q-learning: Multiple Agents 777.1 Problem Formulation and Notation 777.2 Cooperative Detection Using Voronoi Regions and Deep Q-Learning 787.2.1 Agents' Actions and Decision-making 797.2.2 Reactive Decision-making in Voronoi Regions: A Distributed EIG Algorithm 797.2.3 Collective Deep Q-Learning Approach 817.3 Numerical Simulations 887.3.1 Detection of Static Targets 897.3.2 Detection of Moving Targets 927.3.3 Learning Errors and Run Time of the Collective Q-max Algorithm 947.4 Conclusions 958 Conclusions 97References 99Index 103