User-Centered Assessment Design

Madhabi Chatterji, PhD, is Professor Emerita of Measurement, Evaluation, and Education at Teachers College, Columbia University, where she founded the Assessment and Evaluation Research Initiative (AERI). AERI is dedicated to promoting meaningful use of assessment and evaluation information to improve equity and the quality of practices and policies in education, psychology, and the health professions. An award-winning, internationally recognized methodologist and educationist, Dr. Chatterji has taught and mentored numerous doctoral students and postdoctoral researchers over her 30-plus-year career. She is author or editor of more than 100 publications and is a Fellow of the National Education Policy Center. A public intellectual, Dr. Chatterji has spoken out frequently on the limitations of large-scale tests and the adverse social consequences of misused high-stakes educational assessments. Her longstanding scholarly interests lie in instrument design, validation, validity, and test use issues; improving program and policy evaluation designs to support evidence-based practices; and closing learning gaps with proximal diagnostic assessments.

"This text serves as a consolidated resource for technical and practical knowledge regarding assessment design. Excitingly, the author centers issues of score use and consequences in framing the relevance and implications of assessment use. The way Chatterji does a deep dive on each example and then summarizes the key steps will be very helpful to my students."--Robyn Thomas Pitts, PhD, Morgridge College of Education, University of Denver"Chatterji's process model integrates the 'why,' 'who,' 'what,' and 'how-to' of effective assessment This book provides practitioners with a conceptual framework and relevant procedures for conceptualizing and developing carefully targeted measures and establishing their technical adequacy."--Paul Yovanoff, PhD, Simmons School of Education and Human Development (Emeritus), Southern Methodist University"This well-constructed text will be useful for graduate-level courses in testing and measurement. It outlines basic concepts of test construction quite well and presents many figures and applications to make it easier to understand the material."--Matthew K. Burns, PhD, Rose and Irving Fein Endowed Professor of Special Education, University of Florida; Assistant Director, University of Florida Literacy Institute"This book is a 'must have' for those of us in the assessment world. It covers all the basic information that is needed for high-quality assessment development, administration, and analysis. I recommend this book for district- and state-level education decision makers and anyone who provides professional development to program specialists and classroom teachers."--Beverly Fitzpatrick, PhD, School of Pharmacy and School of Education, Memorial University of Newfoundland and Labrador, Canada-

• I. Foundations1. Foundational Concepts in Assessment Design1.1 Chapter Overview1.2 Assessments: Old and Emerging Traditions, a Starting Definition, and Some Distinctions1.3 Viewpoints on Assessment, Measurement, Testing, and Evaluation1.4 Role of Assessment in Scientific, Professional, and Practical Endeavors1.5 Evaluating the Quality of Assessments and Construct Measures: Validity, Reliability, and Utility1.6 Integrating Assessment Design, Validation, and Use: A User-Centered Process1.7 Summary2. Why Assess?: Measure-Based Inferences, Uses, Users, and Consequences2.1 Chapter Overview2.2 Back to the Future: Early Drivers, Milestones, and Consequences of Assessment Use2.3 Modern Drivers and Consequences of Assessment Uses in Education2.4 Modern Drivers and Consequences of Assessment Uses in Psychology, Health, Business, and Other Fields2.5 Applying User-Centered Principles to Improve Practices2.6 Summary3. Whom to Assess? and How?: Specifying the Population and the Assessment Operations3.1 Chapter Overview3.2 Why Population Characteristics and the Socioecological Contexts of Assessments Matter3.3 What Is Measurement Bias?: Case Studies and Hypothetical Illustrations3.4 Selecting Assessment Operations for Diverse Populations and Multidisciplinary Constructs3.5 Steps and Actions: Specifying Whom to Assess? and How? with the Process Model3.6 SummaryII. Assessment Design4. What to Assess?: Specifying the Domains for Constructs4.1 Chapter Overview4.2 Domain Sampling and Domain Specification: Functional Theory and Applied Illustrations4.3 Construct Types, Domain Conceptualizations, and Structures4.4 Domain Specification as a Part of the Process Model: Steps, Techniques, Guidelines, and Conventions4.5 Content-Validating Specified Domains4.6. Summary5. Designing Assessments with Structured and Constructed-Response Items5.1 Chapter Overview5.2 Why the Mechanics of Item Construction Matter5.3 Cognitive Constructs Measured Best with Structured- or Constructed-Response Items5.4 Writing Structured-Response Items: Principles, Guidelines, and Applied Examples5.5 Guidelines for Designing Constructed-Response and Essay Tasks5.6. Instrument Assembly5.7 An Application with the Process Model: A Case Study of Cognitively Based Item and Assessment Design to Foster Learning in Long Division5.8 Summary6. Designing Behavior-Based, Product-Based, and Portfolio-Based Assessments6.1 Chapter Overview6.2 Behavior-, Product-, and Portfolio-Based Assessments: Definitions, Examples, and Origins6.3 Advantages of the Performance Assessment Format6.4 Disadvantages of Performance Assessments: Human Vulnerabilities, Errors, and Biases6.5 Three Case Studies: Applying the Process Model to Design and Validate Performance Assessments6.6 Summary7. Designing Survey-Based and Interview-Based Assessment Tools7.1 Chapter Overview7.2 Self-Report Instruments: Their Defining Properties and Common Applications7.3 Historical Origins of Questionnaires and Attitude Surveys7.4 Measurement Issues with the Self-Report Modality7.5 General Design Guidelines for Self-Report Instruments7.6 Ten More Guidelines for Writing Closed-Ended Survey Items7.7 A Case Study: Applying the Process Model to Design Two Complementary Self-Report Tools7.8 SummaryIII. Validation and Use of Assessments8. Analyzing Data from Assessments: A Statistics Refresher8.1 Chapter Overview8.2 Preparing for Data Analysis8.3 Organizing the Data8.4 Measures of Central Tendency8.5 Measures of Variability8.6 Graphical Displays of Data8.7 The Standard Normal Distribution and Its Applications8.8 Correlation Coefficients and Their Applications8.9 Related Statistical Techniques8.10 Summary9. Improving the Inferential Utility of Assessment Results: Methods and Limitations9.1 Chapter Overview9.2 Frames of Reference and Derived Scores9.3 Using Norms as the Frame of Reference9.4 Using Criterion Scores or Standards as the Frame of Reference9.5 Using Self as the Frame of Reference9.6 Composite Scores9.7 Grouped Scores, Equated Scales, and Linked Tests9.8 Summary10. A Unified Approach to Construct Validity and Validation: Theory to Evidence10.1 Chapter Overview10.2 Construct Validity: An Evolving Concept10.3 Theoretical Foundations of the Unitarian View of Validation10.4 Main Clusters and Types of Validity Evidence10.5 Random Errors of Measurement and Types of Reliability Evidence10.6 Utility of Measures, Assessments, and Assessment Systems10.7 Unified Validation Plans10.8 Chapter Summary11. Empirical Methods of Validation11.1 Chapter Overview11.2 Planning Empirical Validation Studies11.3 Evaluating Item Performance11.4 Examining Fairness and Measurement Bias11.5 Gathering Evidence of Content-Based Validity11.6 Validating Response Processes: The Cognitive Interview11.7 Gathering Correlational Evidence of Validity11.8 Empirical Estimation of Reliability11.9 Methods to Examine Utility11.10 Evaluating the Evidence: The PSQI Case Revisited11.11 Summary12. User-Centered Assessment Design: Revisiting the Principles, Comparisons, and Conclusions12.1 Chapter Overview12.2 Applying the Principles Undergirding the Process Model: A Summary by Section12.3. A User-Centered Design Process: Comparing the Old with the New12.4 Extended Applications of the Process Model12.5 The Process Model Compared to Existing Models of Assessment Design12.6 Connecting the Process Model with the 2014 Standards12.7 SummaryGlossaryReferencesAuthor IndexSubject IndexAbout the Author