Assessing Teacher Competency

Judy R. Wilkerson is an Associate Professor of Research and Assessment at Florida Gulf Coast University, where she teaches graduate and undergraduate courses in measurement and evaluation. As in this book and all of her research, she focuses her efforts with students on providing a highly pragmatic approach, based in theory, with the goal of instilling a commitment in them to assess K-12 learning. Her Ph.D. is in Measurement and Research from the University of South Florida, where she served for 15 years as Director of Program Review, leading College and University efforts in accreditation. Her career has been dedicated to standards-based assessment of programs and teachers, beginning with the creation of an evaluation model for accreditation in 1987, which she implemented in several states. Beginning in 1990 for 15 years, she served as the primary consultant for higher education to the Florida Department of Education, where she drafted the standards for the initial approval of teacher education programs, designed the program approval process, and provided technical assistance to colleges of education in evaluation of teachers and programs statewide. She has consulted nationally on NCATE accreditation and worked with state associations of teacher educators on accreditation related issues. She has also consulted with school districts in Florida on assessment systems for teachers. She was lead designer of the assessment system for the Florida Alternative Certification Program, now used in over 40 of the 68 school districts in the State. William Steve Lang is a Professor of Educational Measurement and Research at the University of South Florida St. Petersburg, where he teaches graduate courses in measurement, statistics, and research.   He, too, focuses his teaching on making meaningful and pragmatic uses of the disciplines he teaches.  He earned his Ph.D. from the University of Georgia in 1984.  He has taught as a public school teacher in South Carolina and Georgia and as a college faculty member in South Carolina, Georgia, and Florida.   He has published on a variety of applications in educational testing and works extensively with the Rasch Model of Item Response Theory.  He began working extensively with Judy when she joined the faculty of the St. Petersburg Campus in 2001.  Since that time, they have collaborated in all aspects of their research and service efforts with the Florida Department of Education, Florida school districts, and teacher education programs nationwide.   They are working together to build two teacher assessment scales – one on teacher competencies, the subject of this book, and another on dispositions.  Their work in both areas is standards-driven.

"Provides possible solutions for the problems faced in the assessment of future teachers, and realistically reveals the extent of the task of teacher certification. It provides those responsible for teacher certification with a structured learning experience that should improve our abilities with this task."

• A Step-by-Step Guide to the CAATS ModelForeword by Richard KunkelPrefaceAcknowledgmentsAbout the Authors1. Expectations and Options for Accountability and Teacher AssessmentWhat This Chapter Is AboutThe Challenge from the National Commission on Teaching and America’s FutureTitle II of the Higher Education Act Amendments of 1998No Child Left Behind LegislationNational Research Council (2001) -- the Committee on Assessment and Teacher QualityWhat A Few Others Have Said: A Brief Review of the Literature on Testing and LicensureStandards: The Roadmap to Accountability and Scientifically-Based Performance AssessmentThe Principal Sets of Standards Governing Our WorkNational and State Pedagogical and Content StandardsUnit Accreditation and Operational StandardsTechnical Standards for Measurement of Teacher CompetencySome Major Threats to Validity in Most Current Assessment SystemsConceptual Frameworks: Pulling It All TogetherNCATE StandardsAERA, APA, & NCMEINTASC Principles: Where NCATE and AERA, APA, & NCME Standards ConvergeMaking Sense of Conceptual FrameworksOur Conceptual Framework: What We ValueAssessment OptionsRecords of Training CompletedTests and Exam ScoresObservations of PerformancePortfolios of Assessable ArtifactsJob-Related Tasks and Work Sample ProductsK-12 Student Work SamplesWrap-UpActivity #1: What’s Happening in Your State and School?Activity #2: Questionnaire for Faculty Views on Competency AssessmentActivity #3: Assessment Belief ScaleActivity #4: Assessment Options2. Portfolios – To Be or Not To Be?? That IS the Question!What This Chapter Is AboutThe Portfolio: Panacea or Pandora’s BoxPortfolios as Certification “Tests”: Lessons from Standards and HistoryAssessment Illiteracy, Paradigms Shifts, and Conflicting PurposesThe Conflict of Formative vs. Summative EvaluationThe Conflict of Program Approval vs. AccreditationThe Conflict of Business or Legislative Personal/Professional Perspectives vs. Accountability Through Title II RequirementsThe Conflict of Academic Freedom vs. AccountabilityThe Conflict of Constructivism vs. PositivismRecommendations for Use of Portfolios in Accountability ContextsTen Recommendations for Assessment System DesignA Recommended, Standards-Based ModelOverview of “Competency Assessment Aligned with Teacher Standards” (CAATS) ModelCAATS Step 1: Define content, purpose, use, and other contextualCAATS Step 2: Develop a valid sampling planCAATS Step 3: Create or update tasks aligned with standards andCAATS Step 4: Design and implement data aggregation tracking and management systemsCAATS Step 5: Ensure credibility and utility of dataWrap-UpA Note about Chapters 8 and 9ActivitiesActivity #1: Review Your FeelingsActivity #2: Thinking about Conflicting ParadigmsActivity #3: Getting Your Action Plan Started3. CAATS Step 1: Assessment Design InputsWhere We Have Been So FarWhat This Chapter Is AboutCAATS Step 1A: Define the Purposes and Uses of the SystemThe Importance of Purpose and UseDifferent Strokes for Different Folks for Different PurposesPurpose and Use in the Accountability ContextMore on Accountability Based Systems: State Program ApprovalCAATS Step 1B: Define the Propositions or Principles that Guide the SystemCAATS Step 1C: Define the Conceptual Framework or Contents of the SystemSo What Is Assessment Content?Standards as the Link between Purpose, Use, and ContentCAATS Step 1D: Review Local Factors That Impact the SystemWrap UpWorksheetsWorksheet #1: Purpose, Use, Propositions, Content, & Context ChecksheetWorksheet #2: Purpose, Use, Content, DraftWorksheet #3: PropositionsWorksheet #4: Contextual Analysis4. CAATS Step 2: Planning with a Continuing Eye on Valid Assessment DecisionsWhere We Have Been So FarWhat This Chapter Is AboutCAATS Step 2A: Organize Standards into Content DomainsAll Those Standards SetsOrganizing for Alignment“A Rose Is a Rose” Or More of the SameWhy Bother?Crosswalks and StandardsCAATS Step 2B: Visualize the Competent Teacher Based on the StandardsCAATS Step 2C: Brainstorm a Set of Summative Tasks (Sampling Plan)CAATS Step 2D: Sort Tasks into Formative and Summative AssessmentsCAATS Step 2E: Build Assessment FrameworksFramework OptionsA Special Case: Aligning Tasks with NCATE RequirementsWrap-UpWorksheetsWorksheet #1: Organizing for Alignment (Version 1)Worksheet #1: Organizing for Alignment (Version 2)Worksheet #2: Our Critical SkillsWorksheet #3: Visualizing the Competent TeacherWorksheet #4: Critical Task ListWorksheet #5: Sorting Formative and Summative TasksWorksheet #6: List of Summative Assessments by Competency TypeWorksheet #7: List of Summative Assessments by Levels of InferenceWorksheet #8: List of Summative Assessments by Points in TimeWorksheet #9: Matrix of Standard by Competency TypeWorksheet #10: Matrix of Critical Tasks by Competency Type and BenchmarkWorksheet #11: Aligning Tasks with NCATE Thematic Portfolios5. CAATS Step 3 -- Writing Tasks Designed to Maximize Validity and ReliabilityWhere We Have Been So FarWhat This Chapter Is AboutCAATS Step 3A: Determine the Task Format for Data AggregationWhat Happens When There Is No Format?Can I Use Percents and Total Points? No, They Don’t Cut It!Elements of a Common Task FormatCAATS Step 3B: Create New Tasks or Modify Existing TasksBasic Concepts about TasksHints and Advice about Writing TasksRubric Examples: A Rose is a RoseCAATS Step 3C: Conduct First Validity StudyCAATS Step 3D: Align Tasks with InstructionWrap-UpWorksheetsWorksheet #1: Proficiency Level DescriptionsWorksheet #2: Task DesignWorksheet #3: Standards and Indicators Coverage ReportWorksheet #4: Individual Task Review for Job-RelatednessWorksheet #5: Checklist for Reviewing Individual TasksWorksheet #6: Instructional Alignment6. CAATS Step 4: Decision-Making and Data ManagementWhere We Have Been So FarWhat This Chapter Is AboutCAATS Step 4A: Determine How Data Will Be AggregatedSo What Is Data Aggregation?The Relationship of Data Aggregation (Step 4A), Cut Scores, (Step 4B), andAn Approach to Decision-Making without Using Points and PercentsCAATS Step 4B: Set Standards for Minimal CompetencyDifferent Strokes for Different Folks – Déjà Vu or Purpose RevisitedIs 100% Really Reasonable?Criticality Yardstick Approach (CYA) to Cut Score Setting in ComplexPerformance AssessmentsFirst CutSecond CutWorking with JudgesAn Example of What to Say to the JudgesCAATS Step 4C: Select and Develop a Tracking SystemSharing Information for Decision-Making: The Big ChallengeData Storage Option #1: Course Grades or Records of Participation/AttendanceData Storage Option #2: Teacher FoldersData Storage Option #3: PortfoliosData Storage Option #4: Electronic Data Management SystemReporting Aggregated DataCAATS Step 4D: Develop Management SystemAdvising and Due ProcessScoring ProceduresImplementationWrap-UpWorksheetsWorksheet #1: Cut Score DecisionsWorksheet #2: Sample Format for Candidate/Teacher Tracking FormWorksheet #3: Format for Data AggregationWorksheet #4: Management PlanWorksheet #5: Rater Monitoring Record7. CAATS Step 5: Credible DataWhere We Have Been So FarWhat This Chapter Is AboutWhat Is Psychometric Integrity and Why Do We Have to Worry About It?CAATS Step 5A: Create a Plan to Provide Evidence of Validity, Reliability, Fairness, and UtilityElements of a PlanElement #1: Purpose and UseElement #2: Construct MeasuredElement #3: Interpretation and Reporting of ScoresElement #4: Assessment Specifications and Content MapElement #5: Assessor/Rater Selection and Training ProceduresElement #6: Analysis MethodologyElement #7: External Review Personnel and MethodologyElement #8: Evidence of Validity, Reliability, and Fairness (VRF)CAATS Step 5B: Implement the Plan ConscientiouslyWrap UpWorksheets and ExamplesWorksheet #1: Assessment SpecificationsWorksheet #2: Analysis of Appropriateness of Decisions for Teacher FailuresWorksheet #3: Program Improvement RecordWorksheet #4: Expert RescoringWorksheet #5: Fairness ReviewWorksheet #6: Analysis of Remediation Efforts and EO ImpactWorksheet #7: Psychometric Plan FormatExample #1 (Empirical Data): Logistic Ruler for Content ValidityExample #2 (Empirical Data): Computation of the Lawshe (1975) ContentExample #3 (Empirical Data): Disparate Impact AnalysisExample #4 (Empirical Data): Computation of Cohen’s Kappa (1960) forExample #5 (Empirical Data): Spearman Correlation Coefficient and Scatterplot:Example #6 (Empirical Data): Pearson Correlation Coefficient and Scatterplot:Example #7 (Empirical Data): Correlation Matrix and Scatterplots Knowledge,Example #8 (Empirical Data): T-Test Comparing Mathematics and ScienceExample #9 (Empirical Data): Differential Item and Person Functioning8. The Trouble with Tribbles: Standard Setting for Professional CertificationWhat This Chapter Is AboutAn Overview of Cut Score SettingFrom Norm-Referenced Objective Tests to Criterion-Referenced Subjective TasksControlling Human JudgmentDifficulty vs. ImportanceSocial ConsequencesTidbits to RememberCut Scores Setting in Traditional, Objective TestsHolistic Impressions: Method OneItem Content: Method TwoPerformance of Examinees: Method ThreeCombination Approach: Method FourStandard or Cut Score Setting in Performance Tests for Professionals: Is it the Same as Multiple Choice?Quotes to RememberThe Extended Angoff Procedure for Performance AssessmentThe Judgmental Policy Capturing Approach for Performance AssessmentThe Dominant Profile Judgment for Performance AssessmentStandard Setting Using Item Response TheoryWhat’s Really Wrong with Current Approaches to Cut Score SettingBerk’s Suggestions and Commentaries Over the YearsThe Criticality Yardstick ApproachWrap UpActivitiesActivity #1: Change the Difficulty of an ItemActivity #2: Change the Criticality of a TaskActivity #3: Replicate a Cut ScoreActivity #4: Take Your Choice9. Using Teacher Scores for Continuous ImprovementWhat This Chapter is AboutReasons Why We Use the Rasch ModelThe Classical ApproachA Quick Overview of Where Rasch Fits Into the Grand Scheme of IRT ModelsRasch: The BasicsGetting StartedDifferences that Item Writers MakeGuttman ScalingA Sample Rasch RulerFrom Pictures to NumbersThe Fit StatisticGain Scores – Real or Imagined?Ratings and RatersLearning More about RaschWrap UpActivitiesActivity #1: Decision Making Tool for Measurement10. Legal IntegrityWhat IF?? A Legal ScenarioPsychometric Issues and Legal ChallengesLegal Issues and PrecedentsEnd NoteAppendix: Tasks Developed for Florida Alternative Certification ProgramGlossaryReferencesIndex