Evaluation Theory, Models, and Applications

DANIEL L. STUFFLEBEAM, PHD, is Distinguished University Professor Emeritus at Western Michigan University, Kalamazoo.CHRIS L. S. CORYN, PHD, is director of the Interdisciplinary PhD in Evaluation (IDPE) program and assistant professor in the Evaluation, Measurement, and Research (EMR) program at Western Michigan University. He is the executive editor of the Journal of MultiDisciplinary Evaluation.

• List of Figures, Tables, and Exhibits xiiiDedication xviiPreface xixAcknowledgments xxiiiThe Author xxvIntroduction xxviiChanges to the First Edition xxviiiIntended Audience xxviiiOverview of the Book’s Contents xxixStudy Suggestions xxxiiPart One: Fundamentals of Evaluation 11 OVERVIEW OF THE EVALUATION FIELD 3What Are Appropriate Objects of Evaluations and Related Subdisciplines of Evaluation? 3Are Evaluations Enough to Control Quality, Guide Improvement, and Protect Consumers? 4Evaluation as a Profession and Its Relationship to Other Professions 4What Is Evaluation? 6How Good Is Good Enough? How Bad Is Intolerable? How Are These Questions Addressed? 17What Are Performance Standards? How Should They Be Applied? 18Why Is It Appropriate to Consider Multiple Values? 20Should Evaluations Be Comparative, Noncomparative, or Both? 21How Should Evaluations Be Used? 21Why Is It Important to Distinguish Between Informal Evaluation and Formal Evaluation? 26How Do Service Organizations Meet Requirements for Public Accountability? 27What Are the Methods of Formal Evaluation? 29What Is the Evaluation Profession, and How Strong Is It? 29What Are the Main Historical Milestones in the Evaluation Field’s Development? 302 EVALUATION THEORY 45General Features of Evaluation Theories 45Theory’s Role in Developing the Program Evaluation Field 47Functional and Pragmatic Bases of Extant Program Evaluation Theory 48AWord About Research Related to Program Evaluation Theory 49Program Evaluation Theory Defined 50Criteria for Judging Program Evaluation Theories 52Theory Development as a Creative Process Subject to Review and Critique by Users 56Status of Theory Development in the Program Evaluation Field 57Importance and Difficulties of Considering Context in Theories of Program Evaluation 58Need for Multiple Theories of Program Evaluation 58Hypotheses for Research on Program Evaluation 59Potential Utility of Grounded Theories 62Potential Utility of Metaevaluations in Developing Theories of Program Evaluation 63Program Evaluation Standards and Theory Development 633 STANDARDS FOR PROGRAM EVALUATIONS 69The Need for Evaluation Standards 71Background of Standards for Program Evaluations 73Joint Committee Program Evaluation Standards 74American Evaluation Association Guiding Principles for Evaluators 80Government Auditing Standards 83Using Evaluation Standards 97Part Two: An Evaluation of Evaluation Approaches and Models 1054 BACKGROUND FOR ASSESSING EVALUATION APPROACHES 107Evaluation Approaches 109Importance of Studying Alternative Evaluation Approaches 109The Nature of Program Evaluation 110Previous Classifications of Alternative Evaluation Approaches 110Caveats 1125 PSEUDOEVALUATIONS 117Background and Introduction 117Approach 1: Public Relations Studies 119Approach 2: Politically Controlled Studies 120Approach 3: Pandering Evaluations 122Approach 4: Evaluation by Pretext 123Approach 5: Empowerment Under the Guise of Evaluation 125Approach 6: Customer Feedback Evaluation 1276 QUASI-EVALUATION STUDIES  133Quasi-Evaluation Approaches Defined 133Functions of Quasi-Evaluation Approaches 134General Strengths and Weaknesses of Quasi-Evaluation Approaches 134Approach 7: Objectives-Based Studies 135Approach 8: The Success Case Method 137Approach 9: Outcome Evaluation as Value-Added Assessment 143Approach 10: Experimental and Quasi-Experimental Studies 147Approach 11: Cost Studies 152Approach 12: Connoisseurship and Criticism 155Approach 13: Theory-Based Evaluation 158Approach 14: Meta-Analysis 1647 IMPROVEMENT- AND ACCOUNTABILITY-ORIENTED EVALUATION APPROACHES 173Improvement- and Accountability-Oriented Evaluation Defined 173Functions of Improvement- and Accountability-Oriented Approaches 174General Strengths and Weaknesses of Decision- and Accountability-Oriented Approaches 174Approach 15: Decision- and Accountability-Oriented Studies 174Approach 16: Consumer-Oriented Studies 181Approach 17: Accreditation and Certification 1848 SOCIAL AGENDA AND ADVOCACY EVALUATION APPROACHES 191Overview of Social Agenda and Advocacy Approaches 191Approach 18: Responsive or Stakeholder-Centered Evaluation 192Approach 19: Constructivist Evaluation 197Approach 20: Deliberative Democratic Evaluation 202Approach 21: Transformative Evaluation 2059 ECLECTIC EVALUATION APPROACHES 213Overview of Eclectic Approaches 213Approach 22: Utilization-Focused Evaluation 214Approach 23: Participatory Evaluation 21910 BEST APPROACHES FOR TWENTY-FIRST-CENTURY EVALUATIONS 229Selection of Approaches for Analysis 230Methodology for Analyzing and Evaluating the Nine Approaches 230Our Qualifications as Raters 230Conflicts of Interest Pertaining to the Ratings 231Standards for Judging Evaluation Approaches 231Comparison of 2007 and 2014 Ratings 236Issues Related to the 2011 Program Evaluation Standards 237Overall Observations 237The Bottom Line 240Part Three: Explication of Selected Evaluation Approaches 24711 EXPERIMENTAL AND QUASI-EXPERIMENTAL DESIGN EVALUATIONS 249Chapter Overview 249Basic Requirements of Sound Experiments 250Prospective Versus Retrospective Studies of Cause 251Uses of Experimental Design 251Randomized Controlled Experiments in Context 252Suchman and the Scientific Approach to Evaluation 256Contemporary Concepts Associatedwith the Experimental andQuasi-Experimental Design Approach to Evaluation 265Exemplars of Large-Scale Experimental and Quasi-Experimental Design Evaluations 269Guidelines for Designing Experiments 271Quasi-Experimental Designs 28012 CASE STUDY EVALUATIONS 291Overview of the Chapter 291Overview of the Case Study Approach 292Case Study Research: The Views of Robert Stake 294Case Study Research: The Views of Robert Yin 297Particular Case Study Information Collection Methods 30113 DANIEL STUFFLEBEAM’S CIPP MODEL FOR EVALUATION: AN IMPROVEMENT AND ACCOUNTABILITY-ORIENTED APPROACH 309Overview of the Chapter 309CIPP Model in Context 309Overview of the CIPP Categories 312Formative and Summative Uses of Context, Input, Process, and Product Evaluations 313Philosophy and Code of Ethics Underlying the CIPP Model 314The Model’s Values Component 317Using the CIPP Framework to Define Evaluation Questions 319Delineation of the CIPP Categories and Relevant Procedures 319Use of the CIPP Model as a Systems Strategy for Improvement 33214 MICHAEL SCRIVEN’S CONSUMER-ORIENTED APPROACH TO EVALUATION  341Overview of Scriven’s Contributions to Evaluation 341Scriven’s Background 343Scriven’s Basic Orientation to Evaluation 343Scriven’s Definition of Evaluation 343Critique of Other Persuasions 344Formative and Summative Evaluation 345Amateur Versus Professional Evaluation 347Intrinsic and Payoff Evaluation 347Goal-Free Evaluation 347Needs Assessment 348Scoring, Ranking, Grading, and Apportioning 349Checklists 352Key Evaluation Checklist 353The Final Synthesis 354Metaevaluation 357Evaluation Ideologies 357Avenues to Causal Inference 361Product Evaluation 363Professionalization of Evaluation 366Scriven’s Look to Evaluation’s Future 36615 ROBERT STAKE’S RESPONSIVE OR STAKEHOLDER-CENTERED EVALUATION APPROACH 373Stake’s Professional Background 374Factors Influencing Stake’s Development of Evaluation Theory 374Stake’s 1967 ‘‘Countenance of Educational Evaluation’’ Article 375Responsive Evaluation Approach 383Substantive Structure of Responsive Evaluation 390Functional Structure of Responsive Evaluation 390An Application of Responsive Evaluation 392Stake’s Recent Rethinking of Responsive Evaluation 39716 MICHAEL PATTON’S UTILIZATION-FOCUSED EVALUATION 403Adherents of Utilization-Focused Evaluation 404Some General Aspects of Patton’s Utilization-Focused Evaluation 405Intended Users of Utilization-Focused Evaluation 407Focusing a Utilization-Focused Evaluation 407The Personal Factor as Vital to an Evaluation’s Success 408The Evaluator’s Roles 408Utilization-Focused Evaluation and Values and Judgments 409Employing Active-Reactive-Adaptive Processes to Negotiate with Users 410Patton’s Eclectic Approach 411Planning Utilization-Focused Evaluations 411Collecting and Analyzing Information and Reporting Findings 412Summary of Premises of Utilization-Focused Evaluation 413Strengths of the Utilization-Focused Evaluation Approach 414Limitations of the Utilization-Focused Evaluation Approach 415Part Four: Evaluation Tasks, Procedures, and Tools 42117 IDENTIFYING AND ASSESSING EVALUATION OPPORTUNITIES 423Sources of Evaluation Opportunities 423Bidders’ Conferences 43118 FIRST STEPS IN ADDRESSING EVALUATION OPPORTUNITIES 435Developing the Evaluation Team 436Developing Thorough Familiarity with the Need for the Evaluation 437Stipulating Standards for Guiding and Assessing the Evaluation 437Establishing Institutional Support for the Projected Evaluation 437Developing the Evaluation Proposal’s Appendix 438Planning for a Stakeholder Review Panel 43919 DESIGNING EVALUATIONS 445A Design Used for Evaluating the Performance Review System of a Military Organization 446Generic Checklist for Designing Evaluations 46220 BUDGETING EVALUATIONS 479Ethical Imperatives in Budgeting Evaluations 480Fixed-Price Budget for Evaluating a Personnel Evaluation System 483Other Types of Evaluation Budgets 486Generic Checklist for Developing Evaluation Budgets 49321 CONTRACTING EVALUATIONS 505Definitions of Evaluation Contracts and Memorandums of Agreement 506Rationale for Evaluation Contracting 508Addressing Organizational Contracting Requirements 511Negotiating Evaluation Agreements 511Evaluation Contracting Checklist 51222 COLLECTING EVALUATIVE INFORMATION 519Key Standards for Information Collection 519An Information Collection Framework 540Useful Methods for Collecting Information 54323 ANALYZING AND SYNTHESIZING INFORMATION 557General Orientation to Analyzing and Synthesizing Information 558Principles for Analyzing and Synthesizing Information 559Analysis of Quantitative Information 560Analysis of Qualitative Information 575Justified Conclusions and Decisions 58024 COMMUNICATING EVALUATION FINDINGS 589Review of Pertinent Analysis and Advice from Previous Chapters 590Complex Needs and Challenges in Reporting Evaluation Findings 591Establishing Conditions to Foster Use of Findings 592Providing Interim Evaluative Feedback 600Preparing and Delivering the Final Report 603Providing Follow-Up Support to Enhance an Evaluation’s Impact 619Part Five: Metaevaluation and Institutionalizing and Mainstreaming Evaluation 62925 METAEVALUATION: EVALUATING EVALUATIONS 631Rationale for Metaevaluation 632Evaluator and Client Responsibilities in Regard to Metaevaluation 634Formative and Summative Metaevaluations 634A Conceptual and Operational Definition of Metaevaluation 634An Instructive Metaevaluation Case 640Metaevaluation Tasks 643Metaevaluation Arrangements and Procedures 647Comparative Metaevaluations 662Checklists for Use in Metaevaluations 664The Role of Context and Resource Constraints 66426 INSTITUTIONALIZING AND MAINSTREAMING EVALUATION 671Review of this Book’s Themes 671Overview of the Remainder of the Chapter 672Rationale and Key Principles for Institutionalizing and Mainstreaming Evaluation 673Early Efforts to Help Organizations Institutionalize Evaluation 674Recent Advances of Use in Institutionalizing and Mainstreaming Evaluation 675Checklist for Use in Institutionalizing and Mainstreaming Evaluation 676Glossary 691References 713Index 744