Pragmatic Software Testing

Rex Black is a 20-plus-year software and systems engineering veteran, and President and Principal Consultant of RBCS, Inc., a leader in software, hardware, and systems testing. He is the author of Managing the Testing Process as well as numerous articles. Black is also President of both the International Software Testing Qualifications Board and the American Software Testing Qualifications Board.

• Acknowledgments xiiiIntroduction xvPart I Goals, Strategies, and Tactics 1Chapter 1 What Does It Mean to Be Pragmatic? 3What Do Effective and Efficient Mean? 3What Effects Do You Want? 4What Is the Right Level of Efficiency? 4Avoiding Redundancy 4Reducing Cost 5What Software Testing Isn’t…But Is Often Thought to Be 6Five Phases of a Tester’s Mental Life 7Other Founding Views on Testing 8Testing as a Form of Quality Risk Management 9So What Is the Test Team All About? 9What Does “Quality” Mean to You? 10Chapter 2 Triangle Test Exercise 13Exercise: The Triangle Test 13Your Solution for the Triangle Test 14Your Solution for the Triangle Test (Continued) 15Author’s Triangle Test Solution 15Chapter 3 Aligning Testing with the Project 19Why Do Organizations Test? 19Perspectives on Testing 20Testing in Context 20Common Test Phases and Objectives 21Testing Throughout the Organization 21The V Model 23Evolutionary and Incremental Models 25The Spiral Model 28Regrettably Common Model: Code and Fix 28Testing Maintenance Releases 29System Integration 30Hardware/Software Development 32The Test Process 32Chapter 4 Understanding Test Strategies, Tactics, and Design 35Aligning Mission, Strategies, and Tactics 35Analytical Test Strategies 36Model-Based Test Strategies 37Methodical Test Strategies 38Process-Oriented Test Strategies 38Dynamic Test Strategies 39Philosophical Test Strategies 40Regression 41Regression Strategy 1: Repeat All Tests 42Regression Strategy 2: Repeat Some Tests 42Three Other Regression Strategies 43Tactics: Categories of Testing Techniques 44Strategic and Tactical Considerations 46The Test System 47Classic Principles for Test Design 50Phases of Test Development 51Synopsis of Test Strategies, Tactics, and Design 52Part II Risk-Based Testing 53Chapter 5 Understanding Risks to System Quality 55Categories of Quality Risks 56Functionality 56Performance and Reliability 57Stress, Capacity, and Volume 59States 59Transactions 60Installation and Deinstallation 61Operations 62Maintenance and Maintainability 63Regression 63Usability and User Interface 64Data Quality 65Error and Disaster Handling and Recovery 66Date and Time Handling 67Localization 68Configuration and Compatibility 69Networked, Internetworked, and Distributed 69Standards and Regulatory Compliance 70Security 70Timing and Coordination 71Documentation 71Can You Think of Other Quality Risks? 72Chapter 6 Aligning Testing with Quality Risks 73Prioritize Risks to System Quality 73Testing, Customer Usage, and System Configurations 74Approaches for Quality Risks Analysis 76Informal Quality Risk Analysis 78Tips for Risk Analysis 81Challenges of Risk Analysis 82Chapter 7 Quality Risk Analysis Exercise 85My Solution 86Comments on My Solution 91A Funny Thing Happened to Me on the Way to the Quality Risks… 92Bonus Exercise 95Template 96My Solution 100Comments on My Solution 101Bonus Example: Grays and Blues Quality Risk Analysis 102Part III Static Testing 105Chapter 8 Reviews and Other Static Tests 107Testing Requirements and Designs 107Reviews Costs and Benefits 108Types of Reviews 109Reviews for Verification and Validation 109Reviews as a Way to Improve the Process, Improve the System, and Reduce Costs 110Answers to the Three Questions about Static Testing 114Reviews as a Way to Achieve Consensus and Understanding 115The Review Process, Roles, and Responsibilities 116Deliverables and Ground Rules from Reviews 117Common Requirements and Design Bugs 118Reviewing (and Testing) Documentation 119Other Static Tests 120Chapter 9 Review Exercise 123Reviews 123Your Solution 124My Solution 128Comments on My Solution 132Bonus Exercise: Reviewing Triangle Requirements Title Page 132Triangle Requirements Review Using Wiegers’s List 132The Requirements Bugs You Found 134My Solution: Requirements Bugs 137Comments on My Solution 139Part IV Behavioral Testing 141Chapter 10 Equivalence Classes and Boundary Values 143Equivalence Partitioning 144Boundary Value Analysis 144Integer 145Real Numbers 147Character and String 149Date 152Time 154Currency 155Beyond Functional Classes and Boundaries 157Chapter 11 Equivalence Classes and Boundary Values Exercise 159Functional Boundaries and Classes 159My Solution and Comments 161Nonfunctional Boundaries and Classes 166Omninet Marketing Requirements Document (v0.3) 166Omninet System Requirements Document (v0.2) 168Chapter 12 Use Cases, Live Data, and Decision Tables 169Use Cases and Scenario Tests 169Nouns and Verbs, Adjectives and Adverbs 172Live Data and Customer Tests 173Decision Tables 174Chapter 13 Decision Table Exercise 177Decision Table Tests 177My Solution and Comments 178Decision Tables and Boundary Values 181My Solution and Comments 183Building a Decision Table for Testing 188My Solution and Comments 190Chapter 14 State Transition Diagrams 197Describing Systems with States Using State Transition Diagrams 197State Tables 199Printer Server State Transition Diagram 200Chapter 15 State Transition Diagram Exercise 203Kiosk States 203My Solution and Comments 204ATM State Models 211My Solution and Comments 212Grays and Blues and State Transition Diagrams 213My Solution and Comments 214Chapter 16 Domain Testing 217Combinatorial Explosions 217A Domain Example Using Frequent-Flyer Programs 218Possible Domain Test Values 220An Aerospace Example 221When Domain Rules Change 223Domain Analysis Summary 224Complex Domain Testing Example 225A General Rule for Complex Domains 228Chapter 17 Domain Testing Exercise 233My Solution and Comments 234Chapter 18 Orthogonal Arrays and All Pairs 239Combinatorial Explosions 239Orthogonal Arrays and All-Pairs Tables 241Two Orthogonal Arrays 241Selecting an Orthogonal Array 243Applying Orthogonal Arrays to a Complicated Real-World Example 244All-Pairs Tables 258Other Thoughts on Configuration Testing 259Chapter 19 Orthogonal Arrays Exercise 261My Solution and Comments 263Chapter 20 Reactive Testing 267General Facts about Reactive Tests 267Error Guessing, Attacks, and Bug Taxonomies 268Bug Hunting 269Exploratory Tests 270Checklists 271Other Sources of Inspiration for Reactive Testing 273Advantages and Disadvantages 274A Case Study of Exploratory Testing 276Part V Structural Testing 279Chapter 21 Control-Flow Testing 281Code Coverage 282McCabe Cyclomatic Complexity 284Chapter 22 Control-Flow Testing Exercise 287My Solution and Comments 288Chapter 23 Data-Flow Testing 293Chapter 24 Data-Flow Testing Exercise 297My Solutions and Comments 298Chapter 25 Integration Testing 307Drivers and Stubs 308Integration Techniques 309Backbone Integration 311McCabe Basis Paths for Integration 313Enhanced Hex Converter Program 316Call Flow 318Chapter 26 Integration Basis Test Exercise 321My Solution and Comments 323Part VI Appendices 325Appendix A Omninet: The Internet Everywhere Marketing Requirements Document 3271 Scope 3291.1 Terms, Acronyms, and Abbreviations 3291.2 Applicable Documents 3302 Required release date 3303 Description of requirements 3303.1 General technical requirements 3303.1.1 Welcome 3303.1.2 Payment 3303.1.3 Internet Browser 3313.1.4 Performance 3313.1.5 Localization 3313.1.6 Content Control 3313.1.7 Session Termination 3313.1.8 Confidentiality 3323.2 Administration 3323.2.1 Software Updates 3323.2.2 View Kiosks 3323.2.3 View Users 3333.2.4 Modify User 3333.2.5 Terminate User 333Appendix B Omninet: The Internet Everywhere System Requirements Document 335Functionality System Requirements 336Reliability System Requirements 340Usability System Requirements 341Efficiency System Requirements 342Maintainability System Requirements 343Portability System Requirements 344Design Models 345Omninet System Architecture 345Payment Processing Decision Table 346Kiosk Module Flow 347Kiosk State-Transition Diagram 348Kiosk State-Transition Table 348Kiosk OS/Browser/Connection Speed Configuration Orthogonal Array 348Appendix C Bibliography and Other Recommended Readings 351RBCS Company Profile 353Index 355