Practice of System and Network Administration, The

Thomas A. Limoncelli is an internationally recognized author, speaker, and system administrator with more than twenty years of experience at companies like Google, Bell Labs, and StackOverflow.com.Christina J. Hogan has more than twenty years of experience in system administration and network engineering, from Silicon Valley to Italy and Switzerland. She has a master’s degree in computer science, a doctorate in aeronautical engineering, and has been part of a Formula 1 racing team.Strata R. Chalup has more than twenty-five years of experience in Silicon Valley, focusing on IT strategy, best-practices, and scalable infrastructures at firms that include Apple, Sun, Cisco, McAfee, and Palm.

• Preface xxxixAcknowledgments xlviiAbout the Authors liPart I: Game-Changing Strategies 1Chapter 1: Climbing Out of the Hole 31.1 Organizing WIP 51.2 Eliminating Time Sinkholes 121.3 DevOps 161.4 DevOps Without Devs 161.5 Bottlenecks 181.6 Getting Started 201.7 Summary 21Exercises 22Chapter 2: The Small Batches Principle 232.1 The Carpenter Analogy 232.2 Fixing Hell Month 242.3 Improving Emergency Failovers 262.4 Launching Early and Often 292.5 Summary 34Exercises 34Chapter 3: Pets and Cattle 373.1 The Pets and Cattle Analogy 373.2 Scaling 393.3 Desktops as Cattle 403.4 Server Hardware as Cattle 413.5 Pets Store State 433.6 Isolating State 443.7 Generic Processes 473.8 Moving Variations to the End 513.9 Automation 533.10 Summary 53Exercises 54Chapter 4: Infrastructure as Code 554.1 Programmable Infrastructure 564.2 Tracking Changes 574.3 Benefits of Infrastructure as Code 594.4 Principles of Infrastructure as Code 624.5 Configuration Management Tools 634.6 Example Infrastructure as Code Systems 674.7 Bringing Infrastructure as Code to Your Organization 714.8 Infrastructure as Code for Enhanced Collaboration 724.9 Downsides to Infrastructure as Code 734.10 Automation Myths 744.11 Summary 75Exercises 76Part II: Workstation Fleet Management 77Chapter 5: Workstation Architecture 795.1 Fungibility 805.2 Hardware 825.3 Operating System 825.4 Network Configuration 845.5 Accounts and Authorization 865.6 Data Storage 895.7 OS Updates 935.8 Security 945.9 Logging 975.10 Summary 98Exercises 99Chapter 6: Workstation Hardware Strategies 1016.1 Physical Workstations 1016.2 Virtual Desktop Infrastructure 1056.3 Bring Your Own Device 1106.4 Summary 113Exercises 114Chapter 7: Workstation Software Life Cycle 1177.1 Life of a Machine 1177.2 OS Installation 1207.3 OS Configuration 1207.4 Updating the System Software and Applications 1237.5 Rolling Out Changes . . . Carefully 1287.6 Disposal 1307.7 Summary 134Exercises 135Chapter 8: OS Installation Strategies 1378.1 Consistency Is More Important Than Perfection 1388.2 Installation Strategies 1428.3 Test-Driven Configuration Development 1478.4 Automating in Steps 1488.5 When Not to Automate 1528.6 Vendor Support of OS Installation 1528.7 Should You Trust the Vendor's Installation? 1548.8 Summary 154Exercises 155Chapter 9: Workstation Service Definition 1579.1 Basic Service Definition 1579.2 Refresh Cycles 1619.3 Tiered Support Levels 1659.4 Workstations as a Managed Service 1689.5 Summary 170Exercises 171Chapter 10: Workstation Fleet Logistics 17310.1 What Employees See 17310.2 What Employees Don't See 17410.3 Configuration Management Database 18310.4 Small-Scale Fleet Logistics 18610.5 Summary 188Exercises 188Chapter 11: Workstation Standardization 19111.1 Involving Customers Early 19211.2 Releasing Early and Iterating 19311.3 Having a Transition Interval (Overlap) 19311.4 Ratcheting 19411.5 Setting a Cut-Off Date 19511.6 Adapting for Your Corporate Culture 19511.7 Leveraging the Path of Least Resistance 19611.8 Summary 198Exercises 199Chapter 12: Onboarding 20112.1 Making a Good First Impression 20112.2 IT Responsibilities 20312.3 Five Keys to Successful Onboarding 20312.4 Cadence Changes 21212.5 Case Studies 21212.6 Summary 216Exercises 217Part III: Servers 219Chapter 13: Server Hardware Strategies 22113.1 All Eggs in One Basket 22213.2 Beautiful Snowflakes 22413.3 Buy in Bulk, Allocate Fractions 22813.4 Grid Computing 23513.5 Blade Servers 23713.6 Cloud-Based Compute Services 23813.7 Server Appliances 24113.8 Hybrid Strategies 24213.9 Summary 243Exercises 244Chapter 14: Server Hardware Features 24514.1 Workstations Versus Servers 24614.2 Server Reliability 24914.3 Remotely Managing Servers 25414.4 Separate Administrative Networks 25714.5 Maintenance Contracts and Spare Parts 25814.6 Selecting Vendors with Server Experience 26114.7 Summary 263Exercises 263Chapter 15: Server Hardware Specifications 26515.1 Models and Product Lines 26615.2 Server Hardware Details 26615.3 Things to Leave Out 27815.4 Summary 278Exercises 279Part IV: Services 281Chapter 16: Service Requirements 28316.1 Services Make the Environment 28416.2 Starting with a Kick-Off Meeting 28516.3 Gathering Written Requirements 28616.4 Customer Requirements 28816.5 Scope, Schedule, and Resources 29116.6 Operational Requirements 29216.7 Open Architecture 29816.8 Summary 302Exercises 303Chapter 17: Service Planning and Engineering 30517.1 General Engineering Basics 30617.2 Simplicity 30717.3 Vendor-Certified Designs 30817.4 Dependency Engineering 30917.5 Decoupling Hostname from Service Name 31317.6 Support 31517.7 Summary 319Exercises 319Chapter 18: Service Resiliency and Performance Patterns 32118.1 Redundancy Design Patterns 32218.2 Performance and Scaling 32618.3 Summary 333Exercises 334Chapter 19: Service Launch: Fundamentals 33519.1 Planning for Problems 33519.2 The Six-Step Launch Process 33619.3 Launch Readiness Review 34519.4 Launch Calendar 34819.5 Common Launch Problems 34919.6 Summary 351Exercises 351Chapter 20: Service Launch: DevOps 35320.1 Continuous Integration and Deployment 35420.2 Minimum Viable Product 35720.3 Rapid Release with Packaged Software 35920.4 Cloning the Production Environment 36220.5 Example: DNS/DHCP Infrastructure Software 36320.6 Launch with Data Migration 36620.7 Controlling Self-Updating Software 36920.8 Summary 370Exercises 371Chapter 21: Service Conversions 37321.1 Minimizing Intrusiveness 37421.2 Layers Versus Pillars 37621.3 Vendor Support 37721.4 Communication 37821.5 Training 37921.6 Gradual Roll-Outs 37921.7 Flash-Cuts: Doing It All at Once 38021.8 Backout Plan 38321.9 Summary 385Exercises 385Chapter 22: Disaster Recovery and Data Integrity 38722.1 Risk Analysis 38822.2 Legal Obligations 38922.3 Damage Limitation 39022.4 Preparation 39122.5 Data Integrity 39222.6 Redundant Sites 39322.7 Security Disasters 39422.8 Media Relations 39422.9 Summary 395Exercises 395Part V: Infrastructure 397Chapter 23: Network Architecture 39923.1 Physical Versus Logical 39923.2 The OSI Model 40023.3 Wired Office Networks 40223.4 Wireless Office Networks 40623.5 Datacenter Networks 40823.6 WAN Strategies 41323.7 Routing 41923.8 Internet Access 42023.9 Corporate Standards 42223.10 Software-Defined Networks 42523.11 IPv6 42623.12 Summary 428Exercises 429Chapter 24: Network Operations 43124.1 Monitoring 43124.2 Management 43224.3 Documentation 43724.4 Support 44024.5 Summary 446Exercises 447Chapter 25: Datacenters Overview 44925.1 Build, Rent, or Outsource 45025.2 Requirements 45225.3 Summary 456Exercises 457Chapter 26: Running a Datacenter 45926.1 Capacity Management 45926.2 Life-Cycle Management 46526.3 Patch Cables 46826.4 Labeling 47126.5 Console Access 47526.6 Workbench 47626.7 Tools and Supplies 47726.8 Summary 480Exercises 481Part VI: Helpdesks and Support 483Chapter 27: Customer Support 48527.1 Having a Helpdesk 48527.2 Offering a Friendly Face 48827.3 Reflecting Corporate Culture 48827.4 Having Enough Staff 48827.5 Defining Scope of Support 49027.6 Specifying How to Get Help 49327.7 Defining Processes for Staff 49327.8 Establishing an Escalation Process 49427.9 Defining “Emergency” in Writing 49527.10 Supplying Request-Tracking Software 49627.11 Statistical Improvements 49827.12 After-Hours and 24/7 Coverage 49927.13 Better Advertising for the Helpdesk 50027.14 Different Helpdesks for Different Needs 50127.15 Summary 502Exercises 503Chapter 28: Handling an Incident Report 50528.1 Process Overview 50628.2 Phase A—Step 1: The Greeting 50828.3 Phase B: Problem Identification 50928.4 Phase C: Planning and Execution 51528.5 Phase D: Verification 51828.6 Perils of Skipping a Step 51928.7 Optimizing Customer Care 52128.8 Summary 525Exercises 527Chapter 29: Debugging 52929.1 Understanding the Customer's Problem 52929.2 Fixing the Cause, Not the Symptom 53129.3 Being Systematic 53229.4 Having the Right Tools 53329.5 End-to-End Understanding of the System 53829.6 Summary 540Exercises 540Chapter 30: Fixing Things Once 54130.1 Story: The Misconfigured Servers 54130.2 Avoiding Temporary Fixes 54330.3 Learn from Carpenters 54530.4 Automation 54730.5 Summary 549Exercises 550Chapter 31: Documentation 55131.1 What to Document 55231.2 A Simple Template for Getting Started 55331.3 Easy Sources for Documentation 55431.4 The Power of Checklists 55631.5 Wiki Systems 55731.6 Findability 55931.7 Roll-Out Issues 55931.8 A Content-Management System 56031.9 A Culture of Respect 56131.10 Taxonomy and Structure 56131.11 Additional Documentation Uses 56231.12 Off-Site Links 56231.13 Summary 563Exercises 564Part VII: Change Processes 565Chapter 32: Change Management 56732.1 Change Review Boards 56832.2 Process Overview 57032.3 Change Proposals 57032.4 Change Classifications 57132.5 Risk Discovery and Quantification 57232.6 Technical Planning 57332.7 Scheduling 57432.8 Communication 57632.9 Tiered Change Review Boards 57832.10 Change Freezes 57932.11 Team Change Management 58132.12 Starting with Git 58332.13 Summary 585Exercises 585Chapter 33: Server Upgrades 58733.1 The Upgrade Process 58733.2 Step 1: Develop a Service Checklist 58833.3 Step 2: Verify Software Compatibility 59133.4 Step 3: Develop Verification Tests 59233.5 Step 4: Choose an Upgrade Strategy 59533.6 Step 5: Write a Detailed Implementation Plan 59833.7 Step 6: Write a Backout Plan 60033.8 Step 7: Select a Maintenance Window 60033.9 Step 8: Announce the Upgrade 60233.10 Step 9: Execute the Tests 60333.11 Step 10: Lock Out Customers 60433.12 Step 11: Do the Upgrade with Someone 60533.13 Step 12: Test Your Work 60533.14 Step 13: If All Else Fails, Back Out 60533.15 Step 14: Restore Access to Customers 60633.16 Step 15: Communicate Completion/Backout 60633.17 Summary 608Exercises 610Chapter 34: Maintenance Windows 61134.1 Process Overview 61234.2 Getting Management Buy-In 61334.3 Scheduling Maintenance Windows 61434.4 Planning Maintenance Tasks 61534.5 Selecting a Flight Director 61634.6 Managing Change Proposals 61734.7 Developing the Master Plan 62034.8 Disabling Access 62134.9 Ensuring Mechanics and Coordination 62234.10 Change Completion Deadlines 62834.11 Comprehensive System Testing 62834.12 Post-maintenance Communication 63034.13 Reenabling Remote Access 63134.14 Be Visible the Next Morning 63134.15 Postmortem 63134.16 Mentoring a New Flight Director 63234.17 Trending of Historical Data 63234.18 Providing Limited Availability 63334.19 High-Availability Sites 63434.20 Summary 636Exercises 637Chapter 35: Centralization Overview 63935.1 Rationale for Reorganizing 64035.2 Approaches and Hybrids 64235.3 Summary 643Exercises 644Chapter 36: Centralization Recommendations 64536.1 Architecture 64536.2 Security 64536.3 Infrastructure 64836.4 Support 65436.5 Purchasing 65536.6 Lab Environments 65636.7 Summary 656Exercises 657Chapter 37: Centralizing a Service 65937.1 Understand the Current Solution 66037.2 Make a Detailed Plan 66137.3 Get Management Support 66237.4 Fix the Problems 66237.5 Provide an Excellent Service 66337.6 Start Slowly 66337.7 Look for Low-Hanging Fruit 66437.8 When to Decentralize 66537.9 Managing Decentralized Services 66637.10 Summary 667Exercises 668Part VIII: Service Recommendations 669Chapter 38: Service Monitoring 67138.1 Types of Monitoring 67238.2 Building a Monitoring System 67338.3 Historical Monitoring 67438.4 Real-Time Monitoring 67638.5 Scaling 68438.6 Centralization and Accessibility 68538.7 Pervasive Monitoring 68638.8 End-to-End Tests 68738.9 Application Response Time Monitoring 68838.10 Compliance Monitoring 68938.11 Meta-monitoring 69038.12 Summary 690Exercises 691Chapter 39: Namespaces 69339.1 What Is a Namespace? 69339.2 Basic Rules of Namespaces 69439.3 Defining Names 69439.4 Merging Namespaces 69839.5 Life-Cycle Management 69939.6 Reuse 70039.7 Usage 70139.8 Federated Identity 70839.9 Summary 709Exercises 710Chapter 40: Nameservices 71140.1 Nameservice Data 71140.2 Reliability 71440.3 Access Policy 72140.4 Change Policies 72340.5 Change Procedures 72440.6 Centralized Management 72640.7 Summary 728Exercises 728Chapter 41: Email Service 72941.1 Privacy Policy 73041.2 Namespaces 73041.3 Reliability 73141.4 Simplicity 73341.5 Spam and Virus Blocking 73541.6 Generality 73641.7 Automation 73741.8 Monitoring 73841.9 Redundancy 73841.10 Scaling 73941.11 Security Issues 74241.12 Encryption 74341.13 Email Retention Policy 74341.14 Communication 74441.15 High-Volume List Processing 74541.16 Summary 746Exercises 747Chapter 42: Print Service 74942.1 Level of Centralization 75042.2 Print Architecture Policy 75142.3 Documentation 75442.4 Monitoring 75542.5 Environmental Issues 75642.6 Shredding 75742.7 Summary 758Exercises 758Chapter 43: Data Storage 75943.1 Terminology 76043.2 Managing Storage 76543.3 Storage as a Service 77243.4 Performance 78043.5 Evaluating New Storage Solutions 78443.6 Common Data Storage Problems 78743.7 Summary 789Exercises 790Chapter 44: Backup and Restore 79344.1 Getting Started 79444.2 Reasons for Restores 79544.3 Corporate Guidelines 79944.4 A Data-Recovery SLA and Policy 80044.5 The Backup Schedule 80144.6 Time and Capacity Planning 80744.7 Consumables Planning 80944.8 Restore-Process Issues 81544.9 Backup Automation 81644.10 Centralization 81944.11 Technology Changes 82044.12 Summary 821Exercises 822Chapter 45: Software Repositories 82545.1 Types of Repositories 82645.2 Benefits of Repositories 82745.3 Package Management Systems 82945.4 Anatomy of a Package 82945.5 Anatomy of a Repository 83345.6 Managing a Repository 83745.7 Repository Client 84145.8 Build Environment 84345.9 Repository Examples 84545.10 Summary 848Exercises 849Chapter 46: Web Services 85146.1 Simple Web Servers 85246.2 Multiple Web Servers on One Host 85346.3 Service Level Agreements 85446.4 Monitoring 85546.5 Scaling for Web Services 85546.6 Web Service Security 85946.7 Content Management 86646.8 Summary 868Exercises 869Part IX: Management Practices 871Chapter 47: Ethics 87347.1 Informed Consent 87347.2 Code of Ethics 87547.3 Customer Usage Guidelines 87547.4 Privileged-Access Code of Conduct 87747.5 Copyright Adherence 87847.6 Working with Law Enforcement 88147.7 Setting Expectations on Privacy and Monitoring 88547.8 Being Told to Do Something Illegal/Unethical 88747.9 Observing Illegal Activity 88847.10 Summary 889Exercises 889Chapter 48: Organizational Structures 89148.1 Sizing 89248.2 Funding Models 89448.3 Management Chain's Influence 89748.4 Skill Selection 89848.5 Infrastructure Teams 90048.6 Customer Support 90248.7 Helpdesk 90448.8 Outsourcing 90448.9 Consultants and Contractors 90648.10 Sample Organizational Structures 90748.11 Summary 911Exercises 911Chapter 49: Perception and Visibility 91349.1 Perception 91349.2 Visibility 92549.3 Summary 933Exercises 934Chapter 50: Time Management 93550.1 Interruptions 93550.2 Follow-Through 93750.3 Basic To-Do List Management 93850.4 Setting Goals 93950.5 Handling Email Once 94050.6 Precompiling Decisions 942