Reliability Engineering in the Cloud

Mariya Breyter is a technology executive, professor, and author of the business bestseller Agile Product and Project Management. With deep expertise in lean and agile methodologies, product management leadership, and enterprise transformation, she has led multi-billion-dollar organizations through innovative cloud strategies, digital transformation, and AI-powered solutions that drive operational excellence, boost efficiency, and elevate customer experience.Carlos Rojas is a technology executive with global experience, a former AWS engineering leader, seasoned expert in resilience and reliability engineering, and AI advisor. His insights stem from decades of software delivery experience across finance, telecom, government, and healthcare IT, transforming teams into industry engineering leaders.

"As someone deeply invested in achieving operational excellence in the cloud, I found this book to be an absolute game-changer. It's a treasure trove of insights for engineering leaders and software teams eager to boost the reliability of their cloud-based systems. The authors write in a clear, engaging style that makes complex concepts easy to grasp. I was particularly impressed by how they illustrate the role of AI in helping organizations anticipate failures, automate responses, and enhance performance. In today's rapidly evolving digital landscape, I genuinely believe this resource is essential for anyone looking to stay ahead of the curve."--Valeria Sadovykh, Technology Strategist, Microsoft"Reliability Engineering in the Cloud is a must-read for anyone aiming to build resilient, scalable, and high-performing cloud systems. With actionable insights, real-world case studies, and strategies leveraging cutting-edge technologies, this book offers a comprehensive guide to ensuring system reliability, optimizing operations and fostering a culture of continuous improvement."--Abhishek Agarwal, author of Product Mastery"Dr. Breyter delivers yet another brilliant work, showcasing her position as a thought leader in enterprise transformation and technical product strategy. Her extensive hands-on experience across diverse industries, mastery of agile methodologies, and commitment to exploring cutting-edge technologies all shine through in this practical book. A must-read for any professional focused on improving service reliability and business results."--Moshe Rasis, management consultant, executive coach, and faculty at New York University"This is an essential guide for building resilient, scalable, and fault-tolerant systems. Covering everything from architecture design and incident response to leveraging gen AI and OKRs, it provides actionable strategies for modern cloud environments. With a focus on automation, observability, and continuous improvement, this book equips teams to master cloud reliability engineering. A must-read for anyone aiming to deliver reliable, high-performing cloud solutions."--Naveen Ks, Agile coach"Whether you're a tech leader or a new engineer, this book provides practical guidance on designing resilient architectures and effective incident response using AI, ML, and gen AI to align with OKRs. Mariya Breyter and Carlos Rojas offer an indispensable resource for achieving operational excellence in the AI-cloud era." --Piyush Sheth, senior lead (Product Enablement, Delivery) at Wells Fargo Helping Enterprises in their Customer-Centric Product Standup, Enabling Innovation, Delivery Journey"Reliability Engineering in the Cloud by Mariya Breyter and Carlos Rojas masterfully integrates AI-driven analytics and Lean methodologies into cloud reliability engineering. This guide bridges theoretical concepts with practical applications, making it essential for those looking to optimize cloud environments. It's an indispensable resource for leaders and engineers aiming to enhance operational excellence and innovation within their organizations. A must-read for mastering cloud reliability."--Srinivasaraju Vysyaraju, senior cybersecurity manager"Reliability Engineering in the Cloud is an indispensable guide for technology leaders managing large-scale, distributed cloud systems. Breyter and Rojas deliver actionable insights on AI-driven observability, fault-tolerant architectures, chaos engineering, and operational automation, providing a comprehensive framework to ensure high availability, scalability, and resilience across mission-critical cloud environments."--Ameesh Paleja, executive vice president, Platform Technology, Capital One"Knowing that strategies fail without proper execution, I highly recommend this book. It equips engineering leaders to establish lean cloud reliability practices. It offers practical frameworks, crucial training, and leadership development techniques that empower cross-functional teams to drive operational excellence and continuous improvement across complex AI-supported cloud environments."--Sara Pendergast, president, Advantage by Design, LLC"An essential guide for enterprise leaders and engineers on how to build reliable and resilient systems to ensure long-term business success in the digital era."--Daria Kirilenko, former senior director, Information Risk Research at Gartner"This book is a must-read for anyone building or leading cloud-based systems. Carlos Rojas, a successful technology executive, provides a comprehensive guide to Cloud Reliability Engineering, offering practical strategies, real-world examples, and cutting-edge technologies to ensure resilient, scalable, and reliable applications. A valuable resource for technology leaders!"--JC Gutierrez, managing director, Technology and Innovation, AWS

• Preface xiiiAcknowledgments xxiiiAbout the Authors xxvChapter 1: Reliability Engineering in the Cloud: How to Design, Build, Operate, and Stress-Test Highly Reliable Systems 1Cloud 1Resilience 2Reliability 2Engineering 3Engineering Excellence 4How to Design and Build Resilient and Reliable Applications 7Leveraging Lean Principles 12Leveraging Artificial Intelligence 13Leveraging Value Stream Mapping 14Culture and Values 15Operational Excellence 15Summary 15Q&A 16Chapter 2: Resilient, Available, and Scalable Systems: Ensuring That Applications Can Handle Failure in a Controlled Manner 21Key Concepts 21Design Principles 33Chaos Engineering 34Validating Resilience 35Summary 39Q&A 40Chapter 3: Incident Response for Fast Recovery: How to Handle Incidents and How to Automate This Process to Improve Time to Detect and Time to Recovery 49Incident Response 49Fast Recovery 55Incident Handling 65Summary 67Q&A 67Chapter 4: Operational Excellence and Change Management: How to Establish Efficient Processes and Maintain Best-in-Class CRE Practices 75Key Performance Indicators 76Root Cause Analysis 76Incident Reviews 78Change Management 83Case Study 86Architecture and Reliability Assessments 89Summary 91Q&A 91Chapter 5: Leveraging Observability, Monitoring, Reliability Metrics, and GenAI: How to Gain Insights, Set Effective Monitoring, Set Service Level Objectives, and Establish Thresholds 95Reliability Engineering Capabilities 95Ten-Step Process for Creating Effective Monitoring 99Maturity Levels 101Monitoring and Alerting Tools 102Case Study: AI’s Impact on CRE 106Summary 108Q&A 109Chapter 6: CRE via Objectives and Key Results (OKRs): How to Build a Culture of Continuous Reliability Improvements Using the OKR Framework 111Continuous Improvement in Lean 111Application of Lean to CRE 115Application of OKRs to CRE 117Summary 122Q&A 122Chapter 7: CRE Tooling: Tools That Support Automatic Failovers, Automatic Rollbacks, Automatic Deployments, Chaos Engineering, Incident Response, Configuration Management, Immutable Infrastructure, and Disaster Recovery 127Distributing Load and Volume with Auto-Scaling and Load Balancing 128Enabling Automatic Failovers for High Availability 133Facilitating Controlled Deployments with Rollback Strategies 136Providing Chaos Engineering Capabilities for Resilience Testing 139Assisting in Incident Response with Automation 140Ensuring Proper Configuration Management 142Leveraging Immutable Infrastructure as a Service 144Practicing Disaster Recovery Frequently 147Case Study 149Summary 151Q&A 151Chapter 8: Cutting-Edge Technologies: How to Use the Power of AI, ML, LLMs, and GenAI Models to Revolutionize Your CRE Practices 155Understanding AI, ML, LLMs, and GenAI 155Benefits of Integrating These Technologies into CRE Practices 156Implementation Considerations 165Summary 169Q&A 169Chapter 9: CRE Value Stream: How to Build Your CRE Strategy Based on Holistic End-to-End Analysis of Your Systems and Customers 175What Is a Value Stream? 175CRE as a Value Stream 176Case Studies 180Summary 182Q&A 182Chapter 10: Culture: How to Build a Psychologically Safe Environment and Culture of Innovation with the CRE Framework 191Psychological Safety 191Employee Empowerment 192Leadership and Ownership 192Collaboration and Cross-Functional Teams 194Customer Obsession 194CRE Culture 195Summary 196Q&A 197Chapter 11: The Business Case for CRE: How to Measure ROI, Ensure Customer Satisfaction, and Promote Business Success 201Benefits of CRE 201Aligning CRE with Strategic Objectives 204Evolution of CRE Practices 206Case Studies 208Summary 209Q&A 210Chapter 12: Conclusion 217Appendix A: Incident Response Checklist Template 219Appendix B: Correction of Error (COE) Document Structure 223Appendix C: CRE Change Management Checklist 225Glossary 227References 231Index 235