Handbook of Automated Scoring

Duanli Yan is Director of Data Analysis and Computational Research in the Psychometrics, Statistics, and Data Sciences area at the Educational Testing Service (ETS), and Adjunct Professor at Fordham University and Rutgers University. She is a co-author of Bayesian Networks in Educational Assessment and Computerized Adaptive and Multistage Testing with R, editor for Practical Issues and Solutions for Computerized Multistage Testing, and co-editor for Computerized Multistage Testing: Theory and Applications. Her awards include the 2016 AERA Division D Significant Contribution to Educational Measurement and Research Methodology Award.André A. Rupp is Research Director in the Psychometrics, Statistics, and Data Sciences area at the Educational Testing Service (ETS). He is co-author and co-editor of two award-winning interdisciplinary books titled Diagnostic Measurement: Theory, Methods, and Applications and The Handbook of Cognition and Assessment: Frameworks, Methodologies, and Applications. His synthesis- and framework-oriented research has appeared in a wide variety of prestigious peer-reviewed journals. He currently serves as the lead developer of the ITEMS professional development portal for NCME. Peter W. Foltz is Vice President in Pearson's AI and Products Solutions Organization and Research Professor at the University of Colorado’s Institute of Cognitive Science. His work covers machine learning and natural language processing for educational and clinical assessments, discourse processing, reading comprehension and writing skills, 21st-century skills learning, and large-scale data analytics. He has authored more than 150 journal articles, book chapters, and conference papers, as well as multiple patents.

'the Handbook of Automated Scoring is an excellent resource for understanding the theoretical, methodological and practical components of automated scoring. It provides a good foundation for understanding the considerations behind how assessments are designed and detailed methodological information about how to best create these kinds of systems. Part 3 that contains different illustrations of how to best design these systems is especially useful for students who are learning more about how these systems should work when implemented correctly.'- Magdalen Beiting-Parrish and Jay Verkuilen, International Statistical Review, 2021

• ContentsForeword..................................................................................................................xi1 The Past, Present, and Future of Automated Scoring..............................1Peter W. Foltz, Duanli Yan, and André A. RuppPart I: Theoretical Foundations2 Cognitive Foundations of Automated Scoring........................................ 13Malcolm I. Bauer and Diego Zapata-Rivera3 Assessment Design with Automated Scoring in Mind.........................29Kristen DiCerbo, Emily Lai, and Matthew Ventura4 Human Scoring with Automated Scoring in Mind................................49Edward W. Wolfe5 Natural Language Processing for Writing and Speaking..................... 69Aoife Cahill and Keelan Evanini6 Multimodal Analytics for Automated Assessment................................93Sidney K. D’Mello7 International Applications of Automated Essay Scoring.................... 113Mark D. Shermis8 Public Perception and Communication around AutomatedEssay Scoring................................................................................................ 133Scott W. Wood9 An Evidentiary-Reasoning Perspective on Automated Scoring:Commentary on Part I................................................................................ 151Robert J. MislevyPart II: Operational Methodologies10 Operational Human Scoring at Scale...................................................... 171Kathryn L. Ricker-Pedley, Susan Hines, and Carolyn Connelly11 System Architecture Design for Scoring and Delivery....................... 195Sue Lottridge and Nick Hoefer12 Design and Implementation for AutomatedScoring Systems........................................................................................... 217Christina Schneider and Michelle Boyer13 Quality Control for Automated Scoring in Large-ScaleAssessment.................................................................................................... 241Dan Shaw, Brad Bolender, and Rick Meisner14 A Seamless Integration of Human and Automated Scoring..............263Kyle Habermehl, Aditya Nagarajan, and Scott Dooley15 Deep Learning Networks for Automated Scoring Applications.......283Saad M. Khan and Yuchi Huang16 Validation of Automated Scoring Systems............................................ 297Duanli Yan and Brent Bridgeman17 Operational Considerations for Automated Scoring Systems:Commentary on Part II............................................................................... 319David M. WilliamsonPart III: Practical Illustrations18 Expanding Automated Writing Evaluation............................................ 329Jill Burstein, Brian Riordan, and Daniel McCaffrey19 Automated Writing Process Analysis.....................................................347Paul Deane and Mo Zhang20 Automated Scoring of Extended Spontaneous Speech.......................365Klaus Zechner and Anastassia Loukina21 Conversation-Based Learning and Assessment Environments.........383Arthur C. Graesser, Xiangen Hu, Vasile Rus, and Zhiqiang Cai22 Automated Scoring in Intelligent Tutoring Systems...........................403Robert J. Mislevy, Duanli Yan, Janice Gobert, and Michael Sao Pedro23 Scoring of Streaming Data in Game-Based Assessments...................423Russell G. Almond24 Automated Scoring in Medical Licensing..............................................445Melissa J. Margolis and Brian E. Clauser25 At the Birth of the Future: Commentary on Part III............................. 469John T. Behrens26 Theory into Practice: Reflections on the Handbook............................. 475Andre A. Rupp, Peter W. Foltz, and Duanli YanGlossary................................................................................................................489References............................................................................................................ 501Index......................................................................................................................553