Reinforcement Learning for Cyber Operations

Dr. Abdul Rahman holds PhDs in physics, math, information technology–cybersecurity and has expertise in cybersecurity, big data, blockchain, and analytics (AI, ML). Dr. Christopher Redino holds a PhD in theoretical physics and has extensive data science experience in every part of the AI / ML lifecycle. Mr. Dhruv Nandakumar has extensive data science expertise in deep learning. Dr. Tyler Cody is an Assistant Research Professor at the Virginia Tech National Security Institute. Dr. Sachin Shetty is a Professor in the Electrical and Computer Engineering Department at Old Dominion University and the Executive Director of the Center for Secure and Intelligent Critical Systems at the Virginia Modeling, Analysis and Simulation Center. Mr. Dan Radke is an Information Security professional with extensive experience in both offensive and defensive cybersecurity.

• List of Figures xvAbout the Authors xixForeword xxiPreface xxiiiAcknowledgments xxvAcronyms xxviiIntroduction xxix1 Motivation 11.1 Introduction 11.1.1 Cyberattack Campaigns via MITRE ATT&CK 41.2 Attack Graphs 41.3 Cyber Terrain 51.4 Penetration Testing 61.5 AI Reinforcement Learning Overview 61.6 Organization of the Book 82 Overview of Penetration Testing 112.1 Penetration Testing 112.2 Importance of Data 432.3 Conclusion 563 Reinforcement Learning: Theory and Application 613.1 An Introduction to Reinforcement Learning (RL) 613.2 RL and Markov Decision Processes 633.3 Learnable Functions for Agents 663.4 Enter Deep Learning 693.5 Q-Learning and Deep Q-Learning 723.6 Advantage Actor-Critic (A2C) 783.7 Proximal Policy Optimization 833.8 Conclusion 854 Motivation for Model-driven Penetration Testing 894.1 Introduction 894.2 Limits of Modern Attack Graphs 914.3 RL for Penetration Testing 934.4 Modeling MDPs 954.5 Conclusion 985 Operationalizing RL for Cyber Operations 1055.1 A High-Level Architecture 1055.2 Layered Reference Model 1075.3 Key Challenges for Operationalizing RL 1135.4 Conclusions 1176 Toward Practical RL for Pen-Testing 1216.1 Current Challenges to Practicality 1216.2 Practical Scalability in RL 1306.3 Model Realism 1366.4 Examples of Applications 1446.5 Realism and Scale 1547 Putting it Into Practice: RL for Scalable Penetration Testing 1617.1 Crown Jewels Analysis 1617.2 Discovering Exfiltration Paths 1657.3 Discovering Command and Control Channels 1717.4 Exposing Surveillance Detection Routes 1767.5 Enhanced Exfiltration Path Analysis 1838 Using and Extending These Models 1938.1 Supplementing Penetration Testing 1938.2 Risk Scoring 1998.3 Further Modeling 2018.4 Generalization 2149 Model-driven Penetration Testing in Practice 2259.1 Recap 2259.2 The Case for Model-driven Cyber Detections 231References 246A Appendix 251Index 253