Designing High Availability Systems

ZACHARY TAYLOR is a Systems Architect at Nokia Solutions & Networks with over thirty years' experience designing high availability and mission critical systems at GE, Lockheed Martin, and Motorola. He has a Masters in Electrical Engineering.SUBRAMANYAM RANGANATHAN is a DFSS Master Black Belt at Nokia Solutions & Networks with over twenty years' experience in the high-tech industry including at Motorola. He has a Masters in Electrical Engineering and an MBA from the Kellogg School of Management.

• Preface xiii List of Abbreviations xvii1. Introduction 12. Initial Considerations for Reliability Design 32.1 The Challenge 32.2 Initial Data Collection 32.3 Where Do We Get MTBF Information? 52.4 MTTR and Identifying Failures 62.5 Summary 73. A Game of Dice: An Introduction to Probability 83.1 Introduction 83.2 A Game of Dice 103.3 Mutually Exclusive and Independent Events 103.4 Dice Paradox Problem and Conditional Probability 153.5 Flip a Coin 213.6 Dice Paradox Revisited 233.7 Probabilities for Multiple Dice Throws 243.8 Conditional Probability Revisited 273.9 Summary 294. Discrete Random Variables 304.1 Introduction 304.2 Random Variables 314.3 Discrete Probability Distributions 334.4 Bernoulli Distribution 344.5 Geometric Distribution 354.6 Binomial Coeffi cients 384.7 Binomial Distribution 404.8 Poisson Distribution 434.9 Negative Binomial Random Variable 484.10 Summary 505. Continuous Random Variables 515.1 Introduction 515.2 Uniform Random Variables 525.3 Exponential Random Variables 535.4 Weibull Random Variables 545.5 Gamma Random Variables 555.6 Chi-Square Random Variables 595.7 Normal Random Variables 595.8 Relationship between Random Variables 605.9 Summary 616. Random Processes 626.1 Introduction 626.2 Markov Process 636.3 Poisson Process 636.4 Deriving the Poisson Distribution 646.5 Poisson Interarrival Times 696.6 Summary 717. Modeling and Reliability Basics 727.1 Introduction 727.2 Modeling 757.3 Failure Probability and Failure Density 777.4 Unreliability, F(t) 787.5 Reliability, R(t) 797.6 MTTF 797.7 MTBF 797.8 Repairable System 807.9 Nonrepairable System 807.10 MTTR 807.11 Failure Rate 817.12 Maintainability 817.13 Operability 817.14 Availability 827.15 Unavailability 847.16 Five 9s Availability 857.17 Downtime 857.18 Constant Failure Rate Model 857.19 Conditional Failure Rate 887.20 Bayes’s Theorem 947.21 Reliability Block Diagrams 987.22 Summary 1078. Discrete-Time Markov Analysis 1108.1 Introduction 1108.2 Markov Process Defined 1128.3 Dynamic Modeling 1168.4 Discrete Time Markov Chains 1168.5 Absorbing Markov Chains 1238.6 Nonrepairable Reliability Models 1298.7 Summary 1409. Continuous-Time Markov Systems 1419.1 Introduction 1419.2 Continuous-Time Markov Processes 1419.3 Two-State Derivation 1439.4 Steps to Create a Markov Reliability Model 1479.5 Asymptotic Behavior (Steady-State Behavior) 1489.6 Limitations of Markov Modeling 1549.7 Markov Reward Models 1549.8 Summary 15510. Markov Analysis: Nonrepairable Systems 15610.1 Introduction 15610.2 One Component, No Repair 15610.3 Nonrepairable Systems: Parallel System with No Repair 16510.4 Series System with No Repair: Two Identical Components 17210.5 Parallel System with Partial Repair: Identical Components 17610.6 Parallel System with No Repair: Nonidentical Components 18310.7 Summary 19211. Markov Analysis: Repairable Systems 19311.1 Repairable Systems 19311.2 One Component with Repair 19411.3 Parallel System with Repair: Identical Component Failure and Repair Rates 20411.4 Parallel System with Repair: Different Failure and Repair Rates 21711.5 Summary 23912. Analyzing Confidence Levels 24012.1 Introduction 24012.2 pdf of a Squared Normal Random Variable 24012.3 pdf of the Sum of Two Random Variables 24312.4 pdf of the Sum of Two Gamma Random Variables 24512.5 pdf of the Sum of n Gamma Random Variables 24612.6 Goodness-of-Fit Test Using Chi-Square 24912.7 Confidence Levels 25712.8 Summary 26413. Estimating Reliability Parameters 26613.1 Introduction 26613.2 Bayes’ Estimation 26813.3 Example of Estimating Hardware MTBF 27313.4 Estimating Software MTBF 27313.5 Revising Initial MTBF Estimates and Tradeoffs 27413.6 Summary 27714. Six Sigma Tools for Predictive Engineering 27814.1 Introduction 27814.2 Gathering Voice of Customer (VOC) 27914.3 Processing Voice of Customer 28114.4 Kano Analysis 28214.5 Analysis of Technical Risks 28414.6 Quality Function Deployment (QFD) or House of Quality 28414.7 Program Level Transparency of Critical Parameters 28714.8 Mapping DFSS Techniques to Critical Parameters 28714.9 Critical Parameter Management (CPM) 28714.10 First Principles Modeling 28914.11 Design of Experiments (DOE) 28914.12 Design Failure Modes and Effects Analysis (DFMEA) 28914.13 Fault Tree Analysis 29014.14 Pugh Matrix 29014.15 Monte Carlo Simulation 29114.16 Commercial DFSS Tools 29114.17 Mathematical Prediction of System Capability instead of “Gut Feel” 29314.18 Visualizing System Behavior Early in the Life Cycle 29714.19 Critical Parameter Scorecard 29714.20 Applying DFSS in Third-Party Intensive Programs 29814.21 Summary 30015. Design Failure Modes and Effects Analysis 30215.1 Introduction 30215.2 What Is Design Failure Modes and Effects Analysis (DFMEA)? 30215.3 Definitions 30315.4 Business Case for DFMEA 30315.5 Why Conduct DFMEA? 30515.6 When to Perform DFMEA 30515.7 Applicability of DFMEA 30615.8 DFMEA Template 30615.9 DFMEA Life Cycle 31215.10 The DFMEA Team 32415.11 DFMEA Advantages and Disadvantages 32715.12 Limitations of DFMEA 32815.13 DFMEAs, FTAs, and Reliability Analysis 32815.14 Summary 33016. Fault Tree Analysis 33116.1 What Is Fault Tree Analysis? 33116.2 Events 33216.3 Logic Gates 33316.4 Creating a Fault Tree 33516.5 Fault Tree Limitations 33916.6 Summary 33917. Monte Carlo Simulation Models 34017.1 Introduction 34017.2 System Behavior over Mission Time 34417.3 Reliability Parameter Analysis 34417.4 A Worked Example 34817.5 Component and System Failure Times Using Monte Carlo Simulations 35917.6 Limitations of Using Nontime-Based Monte Carlo Simulations 36117.7 Summary 36518. Updating Reliability Estimates: Case Study 36718.1 Introduction 36718.2 Overview of the Base Station Controller—Data Only (BSC-DO) System 36718.3 Downtime Calculation 36818.4 Calculating Availability from Field Data Only 37118.5 Assumptions Behind Using the Chi-Square Methodology 37218.6 Fault Tree Updates from Field Data 37218.7 Summary 37619. Fault Management Architectures 37719.1 Introduction 37719.2 Faults, Errors, and Failures 37819.3 Fault Management Design 38119.4 Repair versus Recovery 38219.5 Design Considerations for Reliability Modeling 38319.6 Architecture Techniques to Improve Availability 38319.7 Redundancy Schemes 38419.8 Summary 39520 Application of DFMEA to Real-Life Example 39720.1 Introduction 39720.2 Cage Failover Architecture Description 39720.3 Cage Failover DFMEA Example 39920.4 DFMEA Scorecard 40120.5 Lessons Learned 40220.6 Summary 40321. Application of FTA to Real-Life Example 40421.1 Introduction 40421.2 Calculating Availability Using Fault Tree Analysis 40421.3 Building the Basic Events 40521.4 Building the Fault Tree 40621.5 Steps for Creating and Estimating the Availability Using FTA 40821.6 Summary 41622. Complex High Availability System Analysis 42022.1 Introduction 42022.2 Markov Analysis of the Hardware Components 42022.3 Building a Fault Tree from the Hardware Markov Model 42722.4 Markov Analysis of the Software Components 42722.5 Markov Analysis of the Combined Hardware and Software Components 43322.6 Techniques for Simplifying Markov Analysis 43722.7 Summary 446References 447Index 450