Mastering OpenTelemetry and Observability

STEVE FLANDERS is a Senior Director of Engineering at Splunk, a Cisco company. Steve is one of the founding members of the OpenTelemetry project.

• Foreword xiiiIntroduction xivThe Mastering Series xviChapter 1 What Is Observability? 1Definition 1Background 4Cloud Native Era 4Monitoring Compared to Observability 5Metadata 8Dimensionality 9Cardinality 9Semantic Conventions 10Data Sensitivity 10Signals 10Metrics 10Logs 13Traces 14Other Signals 20Collecting Signals 20Instrumentation 21Push Versus Pull Collection 22Data Collection 23Sampling Signals 26Observability 27Platforms 27Application Performance Monitoring 28The Bottom Line 28Notes 30Chapter 2 Introducing OpenTelemetry! 31Background 31Observability Pain Points 31The Rise of Open Source Software 34Introducing OpenTelemetry 35OpenTelemetry Components 37OpenTelemetry Concepts 48Roadmap 50The Bottom Line 50Notes 51Chapter 3 Getting Started with the Astronomy Shop 53Background 53Architecture 54Prerequisites 54Getting Started 55Accessing the Astronomy Shop 57Accessing Telemetry Data 57Beyond the Basics 58Configuring Load Generation 58Configuring Feature Flags 59Configuring Tests Built from Traces 60Configuring the OTel Collector 60Configuring OTel Instrumentation 62Troubleshooting Astronomy Shop 62Astronomy Shop Scenarios 63Troubleshooting Errors 63Troubleshooting Availability 69Troubleshooting Performance 70Troubleshooting Telemetry 74The Bottom Line 75Notes 76Chapter 4 Understanding the OpenTelemetry Specification 77Background 77API Specification 79API Definition 80API Context 80API Signals 81API Implementation 82SDK Specification 82SDK Definition 83SDK Signals 83SDK Implementation 84Data Specification 84Data Models 86Data Protocols 88Data Semantic Conventions 88Data Compatibility 89General Specification 90The Bottom Line 91Notes 92Chapter 5 Managing the OpenTelemetry Collector 93Background 94Deployment Modes 95Agent Mode 96Gateway Mode 98Reference Architectures 100The Basics 101The Binary 103Sizing 103Components 104Configuration 106Receivers and Exporters 115Processors 116Extensions 126Connectors 127Observing 128Relevant Metrics 128Health Check Extension 131zPages Extension 131Troubleshooting 134Out of Memory Crashes 134Data Not Being Received or Exported 134Performance Issues 135Beyond the Basics 135Distributions 135Securing 137Management 138The Bottom Line 140Notes 141Chapter 6 Leveraging OpenTelemetry Instrumentation 143Environment Setup 144Python Trace Instrumentation 149Automatic Instrumentation 150Manual Instrumentation 157Programmatic Instrumentation 163Mixing Automatic and Manual Trace Instrumentation 166Python Metrics Instrumentation 167Automatic Instrumentation 168Manual Instrumentation 169Programmatic Instrumentation 174Mixing Automatic and Manual Metric Instrumentation 176Python Log Instrumentation 178Manual Metadata Enrichment 179Trace Correlation 181Language Considerations 183NET 184Java 184Go 184Node js 185Deployment Models 185Distributions 185The Bottom Line 186Notes 187Chapter 7 Adopting OpenTelemetry 189The Basics 189Why OTel and Why Now? 190Where to Start? 191General Process 192Data Collection 193Instrumentation 195Production Readiness 196Maturity Framework 197Brownfield Deployment 198Data Collection 198Instrumentation 200Dashboards and Alerts 202Greenfield Deployment 204Data Collection 204Instrumentation 208Other Considerations 208Administration and Maintenance 208Environments 211Semantic Conventions 212The Future 213The Bottom Line 213Notes 214Chapter 8 The Power of Context and Correlation 215Background 215Context 217OTel Context 219Trace Context 221Resource Context 223Logic Context 224Correlation 225Time Correlation 225Context Correlation 226Trace Correlation 228Metric Correlation 230The Bottom Line 230Notes 231Chapter 9 Choosing an Observability Platform 233Primary Considerations 233Platform Capabilities 235Marketing Versus Reality 237Price, Cost, and Value 238Observability Fragmentation 241Primary Factors 242Build, Buy, or Manage 242Licensing, Operations, and Deployment 244OTel Compatibility and Vendor Lock-In 244Stakeholders and Company Culture 245Implementation Basics 246Administration 247Usage 248Maturity Framework 248The Bottom Line 250Notes 250Chapter 10 Observability Antipatterns and Pitfalls 251Telemetry Data Missteps 251Mixing Instrumentation Libraries Scenario 253Automatic Instrumentation Scenario 253Custom Instrumentation Scenario 254Component Configuration Scenario 255Performance Overhead Scenario 255Resource Allocation Scenario 256Security Considerations Scenario 256Monitoring and Maintenance Scenario 257Observability Platform Missteps 258Vendor Lock-in Scenario 260Fragmented Tooling Scenario 260Tool Fatigue Scenario 261Inadequate Scalability Scenario 261Data Overload Scenario 262Company Culture Implications 264Lack of Leadership Support Scenario 265Resistance to Change Scenario 266Collaboration and Alignment Scenario 266Goals and Success Criteria Scenario 267Standardization and Consistency Scenario 268Incentives and Recognition Scenario 268Feedback and Improvement Scenario 269Prioritization Framework 270The Bottom Line 272Notes 273Chapter 11 Observability at Scale 275Understanding the Challenges 275Volume and Velocity of Telemetry Data 276Distributed System Complexity 278Observability Platform Complexity 281Infrastructure and Resource Constraints 281Strategies for Scaling Observability 282Elasticity, Elasticity, Elasticity! 282Leverage Cloud Native Technologies 284Filter, Sample, and Aggregate 286Anomaly Detection and Predictive Analytics 290Emerging Technologies and Methodologies 291Best Practices for Managing Scale 292General Recommendations 292Instrumentation and Data Collection 293Observability Platform 293The Bottom Line 294Notes 295Chapter 12 The Future of Observability 297Challenges and Opportunities 297Cost 297Complexity 299Compliance 300Code 301Emerging Trends and Innovations 302Artificial Intelligence 303Observability as Code 304Service Mesh 305eBPF 306The Future of OpenTelemetry 307Stabilization and Expansion 308Expanded Signal Support 308Unified Query Language 310Community-driven Innovation 310The Bottom Line 311Notes 311Appendix A The Bottom Line 313Chapter 1: What Is Observability? 313Chapter 2: Introducing OpenTelemetry! 315Chapter 3: Getting Started with the Astronomy Shop 316Chapter 4: Understanding the OpenTelemetry Specification 317Chapter 5: Managing the OpenTelemetry Collector 318Chapter 6: Leveraging OpenTelemetry Instrumentation 320Chapter 7: Adopting OpenTelemetry 321Chapter 8: The Power of Context and Correlation 323Chapter 9: Choosing an Observability Platform 324Chapter 10: Observability Antipatterns and Pitfalls 326Chapter 11: Observability at Scale 327Chapter 12: The Future of Observability 328Appendix B Introduction 329Chapter 2: Introducing OpenTelemetry! 330OpenTelemetry Concepts > Roadmap 330Chapter 3: Getting Started with the Astronomy Shop 330Background > Architecture 330Chapter 5: Managing the OpenTelemetry Collector 332Background 332The Basics > Components 332Chapter 12: The Future of Observability 340Challenges and Opportunities > Code 340Notes 341Index 343