AI Product Playbook

Dr. Marily Nika is an award-winning GenAI Product Leader at Google and one of the world's foremost AI educators, with over 13 years of experience building AI products at Google and Meta. She holds a PhD in machine learning and is an author, TED AI speaker, Harvard Business School fellow and co-founder of the AI Product Hub (www.aiproduct.com) which offers AI product management certifications. Diego Granados is a Product Leader with more than 6 years of experience bringing AI products to life in top tech companies in Silicon Valley. He holds an MBA from Duke University and an M.S. in C.S. focused on AI & ML from Georgia Tech and is co-founder of the AI Product Hub (www.aiproduct.com) which offers AI product management certifications.

• Introduction xixPart I Foundational AI/ML Concepts 1Chapter 1 Artificial Intelligence and Machine Learning: What Every Product Manager Needs to Know 3AI vs. ml 4Why This Matters to a PM 4Key Differences Between AI and ml 5Common Misconceptions for PMs: Myths vs. Reality 7Your Glossary as a PM 7Grounding the Concepts: Real-World AI in Action 10The AI PM’s Guiding Principles 14Chapter Summary and Key Takeaways 16Key Takeaways 16Onward: Peeking Under the Hood 17Chapter 2 How Machine Learning Models Learn: A Peek Under the Hood 19The Learning Process: Training, Validation, and Testing 20How Models Learn: An Example with k-Nearest Neighbors (k-NN) 22Applying k-NN (with k=1): 23Another Example: Testing an Unknown Fruit 26Evaluating Model Performance 27The Confusion Matrix: A Foundation for Understanding 27Key Classification Metrics (and Their PM Implications) 28The Precision-Recall Trade-Off 29Choosing the Right Metric 30Overfitting and Underfitting: Striking the Right Balance for Real-World Performance 31Overfitting: Memorizing Instead of Learning 31Underfitting: Missing the Forest for the Trees 32Visual Analogy: Fitting a Curve 32Finding the Sweet Spot: Generalization 33The PM’s Role 33Human-in-the-Loop: Blending AI Power with Human Expertise 34What Is Human-in-the-Loop? 34Why HITL Is Essential for Product Managers (and Their Products) 35How to Implement HITL (PM Considerations) 37Chapter Summary and Key Takeaways 38Key Takeaways 39Onward: Understanding the Broader Process 39Chapter 3 The Big Picture: AI, ML, and You 41Understanding the Relationship Between AI, ML, and Product Goals 41Types of Machine Learning: Understanding the Spectrum of Learning 44Supervised Learning: Guiding the Model with Labeled Examples 46Technical Deep Dive: How Supervised Learning Models Learn from Labeled Data 48Critical Considerations for Product Managers 54Unsupervised Learning: Discovering Hidden Patterns in Your Data 55Technical Deep Dive: How Unsupervised Learning Models Discover Patterns 57Critical Considerations for Product Managers 60Reinforcement Learning: Learning Through Trial and Error 61Technical Deep Dive: How Reinforcement Learning Agents Learn Optimal Policies 63The Learning Process: Exploration, Exploitation, and Q-Learning 65Critical Considerations for Product Managers 67Generative AI: Powering a New Era of Language-Based Applications 67Technical Deep Dive: How LLMs Understand and Generate Language 69Critical Considerations for Product Managers 72The “Gotchas”: A PM’s Guide to LLM Limitations and Risks 73Navigating the Nuances of Generative AI: Understanding GenAI Evaluations— Ensuring Quality and Trust 75Prompt Engineering: The Art and Science of Talking to AI 84Types of Machine Learning: A Recap 89Introduction to Neural Networks and Deep Learning: The Engines of Complex Pattern Recognition 92Neural Networks: Mimicking the Brain’s Connections (But Not Really) 92How Neural Networks Learn: Adjusting the Connections 94Technical Deep Dive: The Mechanics of Neural Networks and Deep Learning 95Challenges in Deep Learning 98Chapter Summary and Key Takeaways 99Key Takeaways 99Onward: Mapping the Process 100Chapter 4 The AI Lifecycle 101Problem Definition and Business Understanding: The “Why” 102Data Collection and Exploration: Understanding Your Ingredients 103Data Preprocessing: Preparing the Ingredients 104Feature Engineering: Crafting the Inputs for Success 104Model Selection and Training: Choosing the Right Algorithm 105Model Evaluation and Tuning: Ensuring Quality 106Model Deployment and Monitoring: Bringing AI to Life (and Keeping It Healthy) 107Retraining and Maintenance: Keeping Your Model Up-to-Date 108Chapter Summary and Key Takeaways 109Key Takeaways 109Onward: Exploring the AI PM Roles 110Part II AI PM Specializations 111Chapter 5 AI-Experiences PM: Shaping User Interaction with AI 113Key Responsibilities: Shaping the AI User Experience 114Day-to-Day Activities 117Required Skills and Knowledge: The AI-Experiences PM Toolkit 120Core Product Management Craft and Practices 120Engineering Foundations for PMs 121Essential Leadership and Collaboration Skills 122AI Lifecycle and Operational Awareness 123Illustrative Example: A Day in the Life of an AI-Experiences PM 124Challenges and Complexities 127How the AI-Experiences PM Interacts with Other Roles 129Chapter Summary and Key Takeaways 134Key Takeaways 134Onward: Architecting the AI Foundation 135Chapter 6 AI-Builder PM: Architecting the Foundation of Intelligent Systems 137Key Responsibilities: Building and Managing the AI Foundation 138Day-to-Day Activities 141Required Skills and Knowledge: The AI-Builder PM’s Technical and Strategic Toolkit 144Core Product Management Craft and Practices 145Engineering Foundations for PMs 146Essential Leadership and Collaboration Skills 147AI Lifecycle and Operational Awareness 148Illustrative Example: A Day in the Life of an AI-Builder PM 149Challenges and Complexities 152How the AI-Builder PM Interacts with Other Roles 154Chapter Summary and Key Takeaways 156Key Takeaways 157Onward: Supercharging the PM Workflow 158Chapter 7 AI-Enhanced PM: Supercharging Product Management with AI 159Key Responsibilities: Augmenting PM Workflows and Decision-Making with AI 160Day-to-Day Activities 162Required Skills and Knowledge: The AI-Enhanced PM’s Toolkit 165Core Product Management Craft and Practices 165Engineering Foundations for PMs 166Essential Leadership and Collaboration Skills 167AI Lifecycle and Operational Awareness 168Illustrative Example: A Day in the Life of an AI-Enhanced PM 169Examples of AI Tools 172Challenges and Complexities 173How the AI-Enhanced PM Interacts with Other Roles 175Skill Comparison: AI-Experiences PM, AI-Builder PM, and AI-Enhanced PM 177Chapter Summary and Key Takeaways 184Key Takeaways 185Onward: From Theory to Action 185Part III Connecting the Dots Between AI/ML Knowledge and PM Craft 187Chapter 8 Identifying and Evaluating AI Opportunities 189Uncovering Potential Use Cases—Mining Your Product for AI Gold 189Recognizing Data-Rich Problem Areas 190Analyzing Existing Data Sources 192Asking the Right Questions 193AI/ML Capability Matching: Connecting Problems to Solutions 194Understanding Your AI/ML Toolkit: Key Capabilities 195Matching Capabilities to Problems: A Practical Approach 200Feature: Search Functionality in a Document Management System 200Feature: Customer Support Chatbot 201Feature: Reporting Dashboard for Marketing Campaigns 201Finding AI Opportunities in the User Journey 202Mapping the User Journey: Charting the Course 202Identifying Pain Points and Opportunities: The AI Detective Work 204Applying AI/ML to Enhance Touchpoints: The Transformation 205Feature Enhancement Through AI/ML— Transforming Existing Functionality 208Identifying Enhancement Opportunities: Finding the Weak Spots 209Applying AI/ML to Enhance Features: The Transformation Process 210Feature: Standard Search Functionality 212Feature: Data Entry Form 212Feature: Reporting Dashboard 212Proactive Product Management—Anticipating User Needs with AI 213Understanding the Power of Prediction and Automation 213Key Areas for Predictive and Automation Opportunities 214Identifying Opportunities: A Practical Approach 216Responsible AI Foundations—Ethical and Feasibility Considerations 217Ethical Considerations: The “Do No Harm” Principle 217Feasibility Considerations: Can We Actually Build This? 220Practical Ideation Techniques for AI/ML Use Cases—Thinking Like an AI-First Product Manager 221Ideation Techniques: Unleashing Your AI Creativity 222“AI Feature Storming”: The Brain Dump 222“AI Scenario Planning”: Walking in the User’s Shoes 223“Data Opportunity Mapping”: Leveraging Your Data Assets 223“AI Capability Alignment”: The Matching Game 224“AI-Powered Feature Reverse Engineering”: Learning from Others 225Cultivating an AI-First Mindset 226Chapter Summary and Key Takeaways 226Key Takeaways 227Onward: Measuring the Value of Your Ideas 227Chapter 9 ROI Calculation for AI Projects: Measuring the Impact and Demonstrating Value 229From Model Performance to Business Impact: A PM’s Guide to AI Metrics 229Defining AI/ML-Specific Metrics: The Foundation for Measuring ROI 230The Importance of Baselines: Knowing Where You Started 230Understanding the Confusion Matrix: Decoding Classification Performance 231Key Performance Metrics for AI/ML Models: Beyond the Confusion Matrix 233Context Matters: Selecting the Right Metrics for Your AI/ML Application 2371. Define Your Business Goals (and Connect Them to User Needs) 2372. Consider the Type of AI/ML Application (and Its Inherent Trade-Offs) 2383. Evaluate the Cost of Errors: The Risk Assessment 2394. Translate Technical Metrics into Business Impact 240Important Considerations 240End-to-End Example—Predicting Churn in a Subscription Service 2411. Identify the Business Goal: Defining the “Why” 2412. Define the AI/ML Application and Solution 2423. Identify Data Sources and Engineer Features: The Raw Materials 243Available Data 243Feature Engineering 243The Product Manager’s Role in This Stage 2444. Select the Metrics: Defining Success 245The Cost of Errors: Prioritizing What Matters 245Our Chosen Metrics 2465. Establish Baseline Metrics: Setting the Starting Point 2466. Conduct Model Training and Evaluation: Building and Testing the AI 2477. Conduct A/B Testing: Measuring Real-World Impact 2488. Calculate the Results and ROI: Quantifying the Value 248Translating Results into Business Impact 249Monitoring for Long-Term Success 2499. Monitor and Maintain the Model for Long-Term Success 250A/B Testing for AI and ML Projects: Validating Impact and Optimizing Performance 251What Is A/B Testing (in a Nutshell)? 251Why Is A/B Testing Especially Important for AI/ML? 252How to Conduct A/B Testing for AI and ML: A Step-by-Step Guide 253Key Considerations for AI/ML A/B Testing 258Chapter Summary and Key Takeaways 259Key Takeaways 259Onward: From the Lab to a Live Product 260Chapter 10 Building and Deploying AI Solutions: From Lab to Live 261MLOps: The Key to Reliable and Scalable AI 261Key Components of MLOps—The AI Production Line 264CI/CD, IaC, and Collaboration: The Foundational Pillars of MLOps 271Glossary of Key MLOps Terms 272MLOps End-to-End Example: Churn Prediction in a Subscription Service (Product Manager’s Perspective) 274Chapter Summary and Key Takeaways 278Key Takeaways 278Onward: Building with Integrity 279Chapter 11 Responsible AI and Ethical Considerations: Building AI with Integrity 281Understanding AI Bias and Fairness: The Foundation of Responsible AI 281Identifying Potential Biases: Where Bias Can Creep In 282Mitigating Potential Biases: A Proactive Approach 285Protected Classes and AI Fairness— Designing for Inclusion 287What are Protected Classes? 287Why Focus on Protected Classes? (The Legal and Ethical Imperative) 287How Protected Classes Relate to AI Bias: The Mechanisms of Discrimination 288Mitigating Bias Related to Protected Classes: Actionable Steps for PMs 289AI Ethics and Legal Compliance—From Principles to Practice 291Understanding the Ethical Landscape: Core Principles 291Understanding the Legal Landscape: Key Regulations 292Actionable Steps for Product Managers: Building Ethically and Legally Compliant AI 293Engaging with the Community and External Stakeholders 297Chapter Summary and Key Takeaways 298Key Takeaways 298Onward: Paving Your Path 299Chapter 12 Conclusion: Paving Your Own Path to AI PM 301Embrace Lifelong Learning: Stay Curious and Iterative 302Cultivate a User-Centric AI Mindset 303Deepen Cross-Functional Collaboration Skills 303Build a Distinct AI Portfolio (Show, Don’t Just Tell) 304Develop a Personal Vision for Your AI Career 305Keep Resilience and Adaptability at the Core 305Final Thoughts 306Index 307