Mark D. Shermis was the principal investigator and academic advisor for the Automated Student Assessment Prize. He is currently Principal for Performance Assessment Analytics, LLC. Dr. Shermis has also held faculty and administrative positions at the University of Houston-Clear Lake, University of Akron, University of Florida, Florida International, Indiana University-Purdue University Indianapolis (IUPUI), and the University of Texas, USA. He is a frequently cited expert on machine scoring and co-author (with Frank DiVesta) of Classroom Assessment in Action.Joshua Wilson is Associate Professor in the School of Education at the University of Delaware, USA. He researches ways that automation and artificial intelligence can improve assessment, teaching, and learning with a specific focus on automated feedback, automated scoring, and automated writing evaluation. Notably, his research has attracted the support of sponsors such as the Institute of Education Sciences of the U.S. Department of Education, the Spencer Foundation, and the Bill and Melinda Gates Foundation.

• Foreword Jill BursteinSection 1: Introduction to AEE and Modern AEE Systems1. Introduction to Automated Evaluation Mark D. Shermis and Joshua Wilson2. Automated Essay Evaluation at Scale: Hybrid Automated Scoring/Hand Scoring in the Summative Assessment ProgramCorey Palermo and Arianto Wibowo3. Exploration of the Stacking Ensemble Learning Algorithm for Automated Scoring of Constructed-Response Items in Reading Assessment Hong Jiao, Shuangshuang Xu, and Manqian Liao4. Scoring Essays Written in Persian Using a Transformer-Based Model: Implications for Multilingual AESTahereh Firoozi and Mark J. Gierl5. SmartWriting-Mandarin: An Automated Essay Scoring System for Chinese Foreign Language LearnersTao-Hsing Chang and Yao-Ting Sung6. NLP Application in the Hebrew Language for Assessment and Learning Yoav Cohen, Anat Ben-Simon, Anat Bar-Siman-Tov, Yona Doleve, Tzur Karelitiz, and Effi LeviSection 2: Expanding Automated Evaluation: Reading, Speech, Mathematics, and Writing Research 7. Automated Scoring for NAEP Short-Form Constructed Responses in Reading Mark D. Shermis8. Automated Scoring and Feedback for Spoken LanguageKlaus Zechner and Ching-Ni Hsieh9. Automated Scoring of Math Constructed-Response Items Scott Hellman, Alejandro Andrade, Kyle Habermehl, Alicia Bouy, and Lee Becker10. We Write Automated Scoring: Using ChatGPT for Scoring in Large-Scale Writing Research ProjectsKausalai (Kay) Wijekumar, Debra McKeown, Shuai Zhang, Pui-Wa Lei, Nikolaus Hruska, and Pablo Pirnay-DummerSection 3: Innovations in Automated Writing Evaluation11. Exploring the Role of Automated Writing Evaluation as a Formative Assessment Tool Supporting Self-Regulated Learning in WritingJoshua Wilson and Charles MacArthur12. Supporting Students’ Text-Based Evidence Use via Formative Automated Writing and Revision AssessmentRip Correnti, Elaine Lin Wang, Lindsay Claire Matsumura, Diane Litman, Zhexiong Liu, and Tianwen Li13. The Use of AWE in Non-English Majors: Student Responses to Automated Feedback and the Impact of Feedback AccuracyAysel Saricaoglu and Zeynep Bilki14. Relationships Between Middle-School Teachers' Perceptions and Application of Automated Writing Evaluation and Student PerformanceAmanda Delgado, Joshua Wilson, Corey Palermo, Tania M. Cruz Cordero, Matthew C. Myers, Halley Eacker, Andrew Potter, Jessica Coles, and Saimou Zhang15. Automated Writing Trait AnalysisPaul Deane16. Advances in Automating Feedback for Argumentative Writing: Feedback Prize as a Case StudyPerpetual Baffour and Scott Crossley17. Automated Feedback in Formative AssessmentHarry A. LaymanSection 4: Factors Affecting the Performance of Automated Evaluation18. Using Automated Scoring to Support Rating Quality Analyses for Human RatersStefanie A. Wind19. Calibrating and Evaluating Automated Scoring Engines and Human Raters over Time Using Measurement ModelsStefanie A. Wind and Yangmeng Xu20. AI Scoring and Writing Fairness Mark D. Shermis21. Automating Bias in Writing Evaluation: Sources, Barriers, and RecommendationsMaria Goldshtein, Amin G. Alhashim, and Rod D. Roscoe22. Explainable AI and AWE: Balancing Tensions between Transparency and Predictive AccuracyDavid Boulanger and Vivekanandan Suresh Kumar23. Validity Argument Roadmap for Automated ScoringDavid Dorsey, Hillary Michaels, and Steve FerraraSection 5: Technological Innovations: "Where Do We Go From Here?" 24. Redesigning Automated Scoring Engines to Include Deep Learning Models Sue Lottridge, Chris Ormerod, and Milan Patel25. Automated Short-Response Scoring for Automated Item Generation in Science Assessments Jinnie Shin and Mark J. Gierl26. Latent Dirichlet Allocation of Constructed ResponsesJordan M. Wheeler, Shiyu Wang, and Allan S. Cohen27. Computational Language as a Window into Cognitive FunctioningPeter W. Foltz and Chelsea Chandler28. Expanding AWE to Incorporate Reading and Writing Evaluation Laura K. Allen, Püren Öncel, and Lauren E. Flynn29. The Two U's in the Future of Automated Essay Evaluation: Universal Access and User-Centered DesignDanielle S. McNamara and Andrew Potter

Mark D. Shermis was the principal investigator and academic advisor for the Automated Student Assessment Prize. He is currently Principal for Performance Assessment Analytics, LLC. Dr. Shermis has also held faculty and administrative positions at the University of Houston-Clear Lake, University of Akron, University of Florida, Florida International, Indiana University-Purdue University Indianapolis (IUPUI), and the University of Texas, USA. He is a frequently cited expert on machine scoring and co-author (with Frank DiVesta) of Classroom Assessment in Action.Joshua Wilson is Associate Professor in the School of Education at the University of Delaware, USA. He researches ways that automation and artificial intelligence can improve assessment, teaching, and learning with a specific focus on automated feedback, automated scoring, and automated writing evaluation. Notably, his research has attracted the support of sponsors such as the Institute of Education Sciences of the U.S. Department of Education, the Spencer Foundation, and the Bill and Melinda Gates Foundation.

• Foreword Jill BursteinSection 1: Introduction to AEE and Modern AEE Systems1. Introduction to Automated Evaluation Mark D. Shermis and Joshua Wilson2. Automated Essay Evaluation at Scale: Hybrid Automated Scoring/Hand Scoring in the Summative Assessment ProgramCorey Palermo and Arianto Wibowo3. Exploration of the Stacking Ensemble Learning Algorithm for Automated Scoring of Constructed-Response Items in Reading Assessment Hong Jiao, Shuangshuang Xu, and Manqian Liao4. Scoring Essays Written in Persian Using a Transformer-Based Model: Implications for Multilingual AESTahereh Firoozi and Mark J. Gierl5. SmartWriting-Mandarin: An Automated Essay Scoring System for Chinese Foreign Language LearnersTao-Hsing Chang and Yao-Ting Sung6. NLP Application in the Hebrew Language for Assessment and Learning Yoav Cohen, Anat Ben-Simon, Anat Bar-Siman-Tov, Yona Doleve, Tzur Karelitiz, and Effi LeviSection 2: Expanding Automated Evaluation: Reading, Speech, Mathematics, and Writing Research 7. Automated Scoring for NAEP Short-Form Constructed Responses in Reading Mark D. Shermis8. Automated Scoring and Feedback for Spoken LanguageKlaus Zechner and Ching-Ni Hsieh9. Automated Scoring of Math Constructed-Response Items Scott Hellman, Alejandro Andrade, Kyle Habermehl, Alicia Bouy, and Lee Becker10. We Write Automated Scoring: Using ChatGPT for Scoring in Large-Scale Writing Research ProjectsKausalai (Kay) Wijekumar, Debra McKeown, Shuai Zhang, Pui-Wa Lei, Nikolaus Hruska, and Pablo Pirnay-DummerSection 3: Innovations in Automated Writing Evaluation11. Exploring the Role of Automated Writing Evaluation as a Formative Assessment Tool Supporting Self-Regulated Learning in WritingJoshua Wilson and Charles MacArthur12. Supporting Students’ Text-Based Evidence Use via Formative Automated Writing and Revision AssessmentRip Correnti, Elaine Lin Wang, Lindsay Claire Matsumura, Diane Litman, Zhexiong Liu, and Tianwen Li13. The Use of AWE in Non-English Majors: Student Responses to Automated Feedback and the Impact of Feedback AccuracyAysel Saricaoglu and Zeynep Bilki14. Relationships Between Middle-School Teachers' Perceptions and Application of Automated Writing Evaluation and Student PerformanceAmanda Delgado, Joshua Wilson, Corey Palermo, Tania M. Cruz Cordero, Matthew C. Myers, Halley Eacker, Andrew Potter, Jessica Coles, and Saimou Zhang15. Automated Writing Trait AnalysisPaul Deane16. Advances in Automating Feedback for Argumentative Writing: Feedback Prize as a Case StudyPerpetual Baffour and Scott Crossley17. Automated Feedback in Formative AssessmentHarry A. LaymanSection 4: Factors Affecting the Performance of Automated Evaluation18. Using Automated Scoring to Support Rating Quality Analyses for Human RatersStefanie A. Wind19. Calibrating and Evaluating Automated Scoring Engines and Human Raters over Time Using Measurement ModelsStefanie A. Wind and Yangmeng Xu20. AI Scoring and Writing Fairness Mark D. Shermis21. Automating Bias in Writing Evaluation: Sources, Barriers, and RecommendationsMaria Goldshtein, Amin G. Alhashim, and Rod D. Roscoe22. Explainable AI and AWE: Balancing Tensions between Transparency and Predictive AccuracyDavid Boulanger and Vivekanandan Suresh Kumar23. Validity Argument Roadmap for Automated ScoringDavid Dorsey, Hillary Michaels, and Steve FerraraSection 5: Technological Innovations: "Where Do We Go From Here?" 24. Redesigning Automated Scoring Engines to Include Deep Learning Models Sue Lottridge, Chris Ormerod, and Milan Patel25. Automated Short-Response Scoring for Automated Item Generation in Science Assessments Jinnie Shin and Mark J. Gierl26. Latent Dirichlet Allocation of Constructed ResponsesJordan M. Wheeler, Shiyu Wang, and Allan S. Cohen27. Computational Language as a Window into Cognitive FunctioningPeter W. Foltz and Chelsea Chandler28. Expanding AWE to Incorporate Reading and Writing Evaluation Laura K. Allen, Püren Öncel, and Lauren E. Flynn29. The Two U's in the Future of Automated Essay Evaluation: Universal Access and User-Centered DesignDanielle S. McNamara and Andrew Potter

Mark D. Shermis was the principal investigator and academic advisor for the Automated Student Assessment Prize. He is currently Principal for Performance Assessment Analytics, LLC. Dr. Shermis has also held faculty and administrative positions at the University of Houston-Clear Lake, University of Akron, University of Florida, Florida International, Indiana University-Purdue University Indianapolis (IUPUI), and the University of Texas, USA. He is a frequently cited expert on machine scoring and co-author (with Frank DiVesta) of Classroom Assessment in Action.Joshua Wilson is Associate Professor in the School of Education at the University of Delaware, USA. He researches ways that automation and artificial intelligence can improve assessment, teaching, and learning with a specific focus on automated feedback, automated scoring, and automated writing evaluation. Notably, his research has attracted the support of sponsors such as the Institute of Education Sciences of the U.S. Department of Education, the Spencer Foundation, and the Bill and Melinda Gates Foundation.

• Foreword Jill BursteinSection 1: Introduction to AEE and Modern AEE Systems1. Introduction to Automated Evaluation Mark D. Shermis and Joshua Wilson2. Automated Essay Evaluation at Scale: Hybrid Automated Scoring/Hand Scoring in the Summative Assessment ProgramCorey Palermo and Arianto Wibowo3. Exploration of the Stacking Ensemble Learning Algorithm for Automated Scoring of Constructed-Response Items in Reading Assessment Hong Jiao, Shuangshuang Xu, and Manqian Liao4. Scoring Essays Written in Persian Using a Transformer-Based Model: Implications for Multilingual AESTahereh Firoozi and Mark J. Gierl5. SmartWriting-Mandarin: An Automated Essay Scoring System for Chinese Foreign Language LearnersTao-Hsing Chang and Yao-Ting Sung6. NLP Application in the Hebrew Language for Assessment and Learning Yoav Cohen, Anat Ben-Simon, Anat Bar-Siman-Tov, Yona Doleve, Tzur Karelitiz, and Effi LeviSection 2: Expanding Automated Evaluation: Reading, Speech, Mathematics, and Writing Research 7. Automated Scoring for NAEP Short-Form Constructed Responses in Reading Mark D. Shermis8. Automated Scoring and Feedback for Spoken LanguageKlaus Zechner and Ching-Ni Hsieh9. Automated Scoring of Math Constructed-Response Items Scott Hellman, Alejandro Andrade, Kyle Habermehl, Alicia Bouy, and Lee Becker10. We Write Automated Scoring: Using ChatGPT for Scoring in Large-Scale Writing Research ProjectsKausalai (Kay) Wijekumar, Debra McKeown, Shuai Zhang, Pui-Wa Lei, Nikolaus Hruska, and Pablo Pirnay-DummerSection 3: Innovations in Automated Writing Evaluation11. Exploring the Role of Automated Writing Evaluation as a Formative Assessment Tool Supporting Self-Regulated Learning in WritingJoshua Wilson and Charles MacArthur12. Supporting Students’ Text-Based Evidence Use via Formative Automated Writing and Revision AssessmentRip Correnti, Elaine Lin Wang, Lindsay Claire Matsumura, Diane Litman, Zhexiong Liu, and Tianwen Li13. The Use of AWE in Non-English Majors: Student Responses to Automated Feedback and the Impact of Feedback AccuracyAysel Saricaoglu and Zeynep Bilki14. Relationships Between Middle-School Teachers' Perceptions and Application of Automated Writing Evaluation and Student PerformanceAmanda Delgado, Joshua Wilson, Corey Palermo, Tania M. Cruz Cordero, Matthew C. Myers, Halley Eacker, Andrew Potter, Jessica Coles, and Saimou Zhang15. Automated Writing Trait AnalysisPaul Deane16. Advances in Automating Feedback for Argumentative Writing: Feedback Prize as a Case StudyPerpetual Baffour and Scott Crossley17. Automated Feedback in Formative AssessmentHarry A. LaymanSection 4: Factors Affecting the Performance of Automated Evaluation18. Using Automated Scoring to Support Rating Quality Analyses for Human RatersStefanie A. Wind19. Calibrating and Evaluating Automated Scoring Engines and Human Raters over Time Using Measurement ModelsStefanie A. Wind and Yangmeng Xu20. AI Scoring and Writing Fairness Mark D. Shermis21. Automating Bias in Writing Evaluation: Sources, Barriers, and RecommendationsMaria Goldshtein, Amin G. Alhashim, and Rod D. Roscoe22. Explainable AI and AWE: Balancing Tensions between Transparency and Predictive AccuracyDavid Boulanger and Vivekanandan Suresh Kumar23. Validity Argument Roadmap for Automated ScoringDavid Dorsey, Hillary Michaels, and Steve FerraraSection 5: Technological Innovations: "Where Do We Go From Here?" 24. Redesigning Automated Scoring Engines to Include Deep Learning Models Sue Lottridge, Chris Ormerod, and Milan Patel25. Automated Short-Response Scoring for Automated Item Generation in Science Assessments Jinnie Shin and Mark J. Gierl26. Latent Dirichlet Allocation of Constructed ResponsesJordan M. Wheeler, Shiyu Wang, and Allan S. Cohen27. Computational Language as a Window into Cognitive FunctioningPeter W. Foltz and Chelsea Chandler28. Expanding AWE to Incorporate Reading and Writing Evaluation Laura K. Allen, Püren Öncel, and Lauren E. Flynn29. The Two U's in the Future of Automated Essay Evaluation: Universal Access and User-Centered DesignDanielle S. McNamara and Andrew Potter