Causal Artificial Intelligence

JUDITH S. HURWITZ is the chief evangelist at Geminos Software, a causal AI platform company. For more than 35 years she has been a strategist, technology consultant to software providers, and a thought leader having authored 10 books in topics ranging from augmented intelligence, data analytics, and cloud computing. JOHN K. THOMPSON is an international technology executive with over 37 years of experience in the fields of data, advanced analytics, and artificial intelligence (AI). John is responsible for the global AI function at EY. He has previously led the global Artificial Intelligence and Rapid Data Lab teams at CSL Behring and is the bestselling author of three books on data analytics.

• Foreword xixPreface xxiiiIntroduction xxixChapter 1 Setting the Stage for Causal AI 1Why Causality Is a Game Changer 2Causal AI in Perspective with Analytics 7Analytical Sophistication Model 8Analytics Enablers 10Analytics 10Advanced Analytics 11Scope of Services to Support Causal AI 11The Value of the Hybrid Team 13The Promise of AI 14Understanding the Core Concepts of Causal AI 15Explainability and Bias Detection 15Explainability 17Detecting Bias in a Model 17Directed Acyclic Graphs 18Structural Causal Model 19Observed and Unobserved Variables 20Counterfactuals 21Confounders 21Colliders 22Front- Door and Backdoor Paths 23Correlation 24Causal Libraries and Tools 25Propensity Score 25Augmented Intelligence and Causal AI 26Summary 27Note 27Chapter 2 Understanding the Value of Causal AI 29Defining Causal AI 30The Origins of Causal AI 33Why Causality? 34Expressing Relationships 37The Ladder of Causation 38Rung 1: Association, or Passive Observation 40Rung 2: Intervention, or Taking Action 40Rung 3: Counterfactuals, or Imagining What If 42Why Causal AI Is the Next Generation of AI 43Deep Learning and Neural Networks 43Neural Networks 44Establishing Ground Truth 45The Business Imperative of a Causal Model 46The Importance of Augmented Intelligence 51The Importance of Data, Visualization, and Frameworks 52Getting the Appropriate Data 52Applying Data and Model Visualization 55Applying Frameworks After Creating a Model 56Getting Started with Causal AI 57Summary 58Notes 59Chapter 3 Elements of Causal AI 61Conceptual Models 62Correlation vs. Causal Models 63Correlation- Based AI 63Causal AI 63Understanding the Relationship Between Correlation and Causality 64Process Models 66Correlation- Based AI Process Model 67Causal- Based AI Process Model 69Collaboration Between Business and Analytics Professionals 72The Fundamental Building Blocks of Causal AI Models 75The Relations Between DAGs and SCMs 76Explaining DAGs 76Causal Notation: The Language of DAGs 78Operationalizing a DAG with an SCM 79The Elements of Visual Modeling 81Nodes 83Variables 83Endogenous and Exogenous Variables 83Observed and Unobserved Variables 84Paths/Relationships 84Weights 86Summary 88Notes 89Chapter 4 Creating Practical Causal AI Models and Systems 91Understanding Complex Models 92Causal Modeling Process: Part 1 94Step 1: What Are the Intended Outcomes? 95Step 2: What Are the Proposed Interventions? 97Step 3: What Are the Confounding Factors? 99Step 4: What Are the Factors Creating the Effects and Changes? 102Common/Universal Effects in a Causal Model 102Refined Effects in a Causal Model 103Step 5: Creating a Directed Acyclic Graph 105Step 6: Paths and Relationships 105Types of Paths 106Path Connecting an Unobserved Variable 107Front- Door Paths 108Backdoor Paths 108Modeling for Simplicity to Understand Complexity 109Step 7: Data Acquisition 110Causal- Based Approach: Part 2 112Step 8: Data Integration 113Step 9: Model Modification 114Step 10: Data Transformation 115Step 11: Preparing for Deployment in Business 118Summary 121Notes 122Chapter 5 Creating a Model with a Hybrid Team 125The Hybrid Team 126Why a Hybrid Team? 127The Benefits of a Hybrid Team 128Establishing the Hybrid Team as a Center of Excellence 129How Teams Collaborate 131But Why? 132Defining Roles 134Leaders and Business Strategists 137Subject- Matter Experts 138Data Experts 140Software Developers 142Business Process Analysts 143Information Technology Expertise 143Project Manager(s) 144The Basics Steps for a Hybrid Team Project 145An Overview of Model Creation 146It Depends on Your Destination 150Understanding the Root Cause of a Problem 151Understanding What Happened and Why 153The Importance of the Iterative Process 154Summary 155Chapter 6 Explainability, Bias Detection, and AI Responsibility in Causal AI 157Explainability 158The Ramifications of the Lack of Explainability 159What Is Explainable AI in Causal AI Models? 161Black Boxes 162Internal Workings of Black-Box Models 162Deep Learning at the Heart of Black Boxes 163Is Code Understandable? 163The Value of White-Box Models 166Understanding Causal AI Code 167Techniques for Achieving Explainability 169Challenges of Complex Causal Models 169Methods for Understanding and Explaining Complex Causal AI Models 171The Importance of the SHAP Explainability Method 172Detecting Bias and Ensuring Responsible AI 175Bias in Causal AI Systems 176Responsible AI: Trust and Fairness 178How Causal AI Addresses Bias Detection 180Tools for Assessing Fairness and Bias 182The Human Factor in Bias Detection and Responsible AI 183Summary 184Note 184Chapter 7 Tools, Practices, and Techniques to Enable Causal AI 185The Causal AI Pipeline 187Define Business Objectives 190Model Development 193Data Identification and Collection 195Data Privacy, Governance, and Security 197Synthetic Data 198Model Validation 199Deployment/Production 201Monitor and Evaluate 203Update and Iterate 205Continuous Learning 208The Importance of Synthetic Data 210Why Create Synthetic Data? 210Overcoming Data Limitations 211Enhancing Data Privacy and Security 211Model Validation and Testing 211Expanding the Range of Possible Scenarios 212Reducing the Cost of Data Collection 212Improving Data Imbalance 213Encouraging Collaboration and Openness 213Streamlining Data Preprocessing 213Supporting Counterfactual Analysis 213Fostering Innovation and Experimentation 214Creating Synthetic Data 214Generative Models 214Agent-Based Modeling 215Data Augmentation 215Data Synthesis Tools and Platforms 215Conditional Synthetic Data Generation 216Synthetic Data from Text 216The Limitations of Synthetic Data 217Current State of Tools and Software in Causal AI 218The Role of Open Source in Causal AI 218Commercial Causal AI Software 221CausaLens 221Geminos Software 223Summary 223Chapter 8 Causal AI in Action 225Enterprise Marketing in a Business- to- Consumer Scenario 226DDCo Marketing Causal Model: Annual Pricing Review and Update Cycle 228Incorporating Internal and External Factors in the Model and DAG 230Easily Enabling Iterating 231End-User-Driven Exploration 232Bench Testing 234DDCo Marketing Causal Model: Semiannual Product Planning Cycle 236Always Consider Model Reuse 237Give and Take in Building a New Model 239Typical Model and Process Operation: Iterating 239Keeping the Process/Model Scope Manageable and Understandable 240Moving from Strategy to Building and Implementing Causal AI Solutions 241Agriculture: Enhancing Crop Yield 242Key Causal Variables 244Creating the DAG 246Moving from the DAG to Implementing the Causal AI Model 247Commercial Real Estate: Valuing Warehouse Space 250Key Causal Variables 251Implementing the Causal AI Model 253Video Streaming: Enhancing Content Recommendations 254Key Causal Variables 255Implementing the Causal AI Model 256Healthcare: Reducing Infant Mortality 258Key Causal Variables 259Implementing the Causal AI Model 261Retail: Providing Executives Actionable Information 263Key Causal Variables 264Implementing the Causal Model 265Summary 267Chapter 9 The Future of Causal AI 271Where We Stand Today 271Foundations of Causal AI 273The Causal AI Journey 274Causal AI Today 274What’s Next for Causal AI 276Integrating Causal AI and Traditional AI 278The Imperative for Managing Data 279Ensembles of Data 279Generative AI Is Emerging as a Game Changer for Causal AI 281The Future of Causal Discovery 282The Emergence of Causal AI Reinforcement Learning Will Accelerate Model Training 284Causal AI as a Common Language Between Business Leaders and Data Scientists 284The Emergence of Probabilistic Programming Languages 286The Predictable Model Evolution Cycle 286The Emergence of the Digital Twin 287Improving the Ability to Understand Ground Truth 289The Development of More Sophisticated DAGs 289Visualizing Complex Relationships in the DAGs 290The Merging of Causal and Traditional AI Models 291The Future of Explainability 291The Evolution of Responsible AI 292Advances in Data Security and Privacy 293Integration Will Be Between Models and Business Applications 294Summary 295Glossary 299Appendix 313Selected Resources 329Acknowledgments 331About the authors 335About the contributor 339Index 341