Experimentation in Software Engineering

'This is a well written and concise book, which provides the reader with the essentials necessary to design, conduct and analyse a software engineering experiment. Its value lies in that it is specifically written for the software engineering field, and has surveyed the major contributions by a number of leading researchers in this area. A 'how to' book is always welcome, both as a useful starting point to the inexperienced and as a helpful reference and reminder of best practice to others.' Software Testing Verification and Reliability, 11 (2001)

• 1 Introduction.- 1.1 Software engineering context.- 1.2 Science and software engineering.- 2 Empirical strategies.- 2.1 Overview of empirical strategies.- 2.2 Surveys.- 2.3 Case studies.- 2.4 Experiments.- 2.5 Empirical strategies comparison.- 2.6 Empiricism in a software engineering context.- 3 Measurement.- 3.1 Basic concepts.- 3.2 Measurements in software engineering.- 4 Experiment process.- 4.1 Variables, treatments, objects and subjects.- 4.2 Process.- 5 Definition.- 5.1 Define experiment.- 5.2 Example.- 5.3 Summary.- 6 Planning.- 6.1 Context selection.- 6.2 Hypothesis formulation.- 6.3 Variables selection.- 6.4 Selection of subjects.- 6.5 Experiment design.- 6.6 Instrumentation.- 6.7 Validity evaluation.- 6.8 Detailed description of validity threats.- 6.9 Priority among types of validity threats.- 7 Operation.- 7.1 Preparation.- 7.2 Execution.- 7.3 Data validation.- 8 Analysis and interpretation.- 8.1 Descriptive statistics.- 8.2 Data set reduction.- 8.3 Hypothesis testing.- 9 Presentation and package.- 9.1 An experiment report outline.- 10 Literature survey.- 10.1 Inspection experiments.- 10.2 Other experiments in Software Engineering.- 10.3 Resources.- 11 Example: Experiment process.- 11.1 Definition.- 11.2 Planning.- 11.3 Operation.- 11.4 Analysis and interpretation.- 11.5 Summary and conclusions.- 12 Example: C versus C++.- 12.1 Introduction and problem statement.- 12.2 Experiment planning.- 12.3 Analysis and interpretation.- 12.4 Conclusions and further work.- 13 Exercises.- 13.1 Understanding.- 13.2 Training.- 13.3 Reviewing.- 13.4 Assignments.- Appendix A: Statistical tables.- Appendix B: Experiment process overview.- References.- About the authors.