Software Testing

ALI MILI is Professor at New Jersey Institute of Technology, USA. His research is focused on software engineering, including technical and organizational aspects, and on software engineering education. FAIROUZ TCHIER is Associate Professor at King Saud University, Saudi Arabia. Her main areas of research are discrete mathematics, theoretical computer science, software engineering, and fuzzy theory.

• Preface xivPart I Introduction to Software Testing 11 Software Engineering: A Discipline Like No Other 31.1 A Young, Restless Discipline 31.2 An Industry Under Stress 51.3 Large, Complex Products 51.4 Expensive Products 71.5 Absence of Reuse Practice 91.6 Fault-Prone Designs 91.7 Paradoxical Economics 101.7.1 A Labor-Intensive Industry 101.7.2 Absence of Automation 111.7.3 Limited Quality Control 111.7.4 Unbalanced Lifecycle Costs 121.7.5 Unbalanced Maintenance Costs 121.8 Chapter Summary 131.9 Bibliographic Notes 132 Software Quality Attributes 142.1 Functional Attributes 152.1.1 Boolean Attributes 152.1.2 Statistical Attributes 152.2 Operational Attributes 172.3 Usability Attributes 182.4 Business Attributes 192.5 Structural Attributes 202.6 Chapter Summary 212.7 Exercises 212.8 Bibliographic Notes 223 A Software Testing Lifecycle 233.1 A Software Engineering Lifecycle 233.2 A Software Testing Lifecycle 273.3 The V-Model of Software Testing 323.4 Chapter Summary 333.5 Bibliographic Notes 34Part II Foundations of Software Testing 354 Software Specifications 374.1 Principles of Sound Specification 384.1.1 A Discipline of Specification 384.2 Relational Mathematics 394.2.1 Sets and Relations 394.2.2 Operations on Relations 394.2.3 Properties of Relations 414.3 Simple Input Output Programs 424.3.1 Representing Specifications 424.3.2 Ordering Specifications 464.3.3 Specification Generation 484.3.4 Specification Validation 534.4 Reliability Versus Safety 604.5 State-based Systems 614.5.1 A Relational Model 624.5.2 Axiomatic Representation 644.5.3 Specification Validation 704.6 Chapter Summary 724.7 Exercises 724.8 Problems 764.9 Bibliographic Notes 785 Program Correctness and Verification 795.1 Correctness: A Definition 805.2 Correctness: Propositions 835.2.1 Correctness and Refinement 835.2.2 Set Theoretic Characterizations 855.2.3 Illustrations 865.3 Verification 885.3.1 Sample Formulas 895.3.2 An Inference System 915.3.3 Illustrative Examples 945.4 Chapter Summary 985.5 Exercises 995.6 Problems 1005.7 Bibliographic Notes 1006 Failures, Errors, and Faults 1016.1 Failure, Error, and Fault 1016.2 Faults and Relative Correctness 1036.2.1 Fault, an Evasive Concept 1036.2.2 Relative Correctness 1046.3 Contingent Faults and Definite Faults 1076.3.1 Contingent Faults 1076.3.2 Monotonic Fault Removal 1096.3.3 A Framework for Monotonic Fault Removal 1146.3.4 Definite Faults 1146.4 Fault Management 1166.4.1 Lines of Defense 1166.4.2 Hybrid Validation 1186.5 Chapter Summary 1216.6 Exercises 1226.7 Problems 1236.8 Bibliographic Notes 1247 A Software Testing Taxonomy 1257.1 The Trouble with Hyphenated Testing 1257.2 A Classification Scheme 1267.2.1 Primary Attributes 1277.2.2 Secondary Attributes 1317.3 Testing Taxonomy 1367.3.1 Unit-Level Testing 1367.3.2 System-Level Testing 1387.4 Exercises 1397.5 Bibliographic Notes 140Part III Test Data Generation 1418 Test Generation Concepts 1438.1 Test Generation and Target Attributes 1438.2 Test Outcomes 1468.3 Test Generation Requirements 1488.4 Test Generation Criteria 1528.5 Empirical Adequacy Assessment 1558.6 Chapter Summary 1608.7 Exercises 1618.8 Bibliographic Notes 1628.9 Appendix: Mutation Program 1639 Functional Criteria 1659.1 Domain Partitioning 1659.2 Test Data Generation from Tabular Expressions 1719.3 Test Generation for State Based Systems 1769.4 Random Test Data Generation 1849.5 Tourism as a Metaphor for Test Data Selection 1889.6 Chapter Summary 1909.7 Exercises 1909.8 Bibliographic Notes 19210 Structural Criteria 19310.1 Paths and Path Conditions 19410.1.1 Execution Paths 19410.1.2 Path Functions 19610.1.3 Path Conditions 20110.2 Control Flow Coverage 20210.2.1 Statement Coverage 20210.2.2 Branch Coverage 20410.2.3 Condition Coverage 20710.2.4 Path Coverage 20910.3 Data Flow Coverage 21410.3.1 Definitions and Uses 21410.3.2 Test Generation Criteria 21710.3.3 A Hierarchy of Criteria 22010.4 Fault-Based Test Generation 22010.4.1 Sensitizing Faults 22110.4.2 Selecting Input Data for Fault Sensitization 22510.4.3 Selecting Input Data for Error Propagation 22710.5 Chapter Summary 22810.6 Exercises 22910.7 Bibliographic Notes 232Part IV Test Deployment and Analysis 23311 Test Oracle Design 23511.1 Dilemmas of Oracle Design 23511.2 From Specifications to Oracles 23811.3 Oracles for State-Based Products 24211.3.1 From Axioms to Oracles 24311.3.2 From Rules to Oracles 24411.4 Chapter Summary 25011.5 Exercises 25112 Test Driver Design 25312.1 Selecting a Specification 25312.2 Selecting a Process 25512.3 Selecting a Specification Model 25712.3.1 Random Test Generation 25712.3.2 Pre-Generated Test Data 26312.3.3 Faults and Fault Detection 26612.4 Testing by Symbolic Execution 26912.5 Chapter Summary 27412.6 Exercises 27512.7 Bibliographic Notes 27913 Test Outcome Analysis 28013.1 Logical Claims 28113.1.1 Concrete Testing 28113.1.2 Symbolic Testing 28213.1.3 Concolic Testing 28313.2 Stochastic Claims: Fault Density 28413.3 Stochastic Claims: Failure Probability 28713.3.1 Faults are Not Created Equal 28713.3.2 Defining/Quantifying Reliability 28913.3.3 Modeling Software Reliability 29113.3.4 Certification Testing 29413.3.5 Reliability Estimation and Reliability Improvement 29513.3.6 Reliability Standards 29913.3.7 Reliability as an Economic Function 30013.4 Chapter Summary 30713.5 Exercises 30813.6 Problems 31013.7 Bibliographic Notes 310Part V Management of Software Testing 31114 Metrics for Software Testing 31314.1 Fault Proneness 31414.1.1 Cyclomatic Complexity 31514.1.2 Volume 31614.2 Fault Detectability 31714.3 Error Detectability 32014.4 Error Maskability 32314.5 Failure Avoidance 32414.6 Failure Tolerance 32614.7 An Illustrative Example 32714.7.1 Cyclomatic Complexity 32714.7.2 Volume 32814.7.3 State Redundancy 32814.7.4 Functional Redundancy 32814.7.5 Non-injectivity 32914.7.6 Non-determinacy 32914.7.7 Summary 33014.8 Chapter Summary 33014.9 Exercises 33114.10 Bibliographic Notes 33215 Software Testing Tools 33315.1 A Classification Scheme 33315.2 Scripting Tools 33415.2.1 CppTest 33415.2.2 SilkTest 33515.3 Record-and-Replay Tools 33615.3.1 TestComplete 33615.3.2 Selenium IDE 33715.4 Performance-Testing Tools 33815.4.1 LoadRunner 33815.4.2 Grinder 33915.4.3 QF-Test 34015.4.4 Appvance PerformanceCloud 34015.4.5 JMeter 34115.5 Oracle Design Tools 34215.5.1 JUnit 34215.5.2 TestNG 34315.6 Exception Discovery 34315.6.1 Rational Purify 34315.6.2 Astree 34415.7 Collaborative Tools 34515.7.1 FitNesse 34515.8 Chapter Summary 34516 Testing Product Lines 34716.1 PLE: A Streamlined Reuse Model 34716.2 Testing Issues 35116.3 Testing Approaches 35316.4 Illustration 35416.4.1 Domain Analysis 35416.4.2 Domain Modeling 35616.4.3 A Reference Architecture 35916.4.4 Domain Implementation 36016.4.5 Testing at Domain Engineering 36516.4.6 Testing at Application Engineering 36916.5 Chapter Summary 37216.6 Exercises 37216.7 Problems 37216.8 Bibliographic References 373Bibliography 374Index 377