Assessing Teacher Dispositions

Judy R. Wilkerson is an Associate Professor of Research and Assessment at Florida Gulf Coast University, where she teaches graduate and undergraduate courses in measurement and evaluation. As in this book and all of her research, she focuses her efforts with students on providing a highly pragmatic approach, based in theory, with the goal of instilling a commitment in them to assess K-12 learning. Her Ph.D. is in Measurement and Research from the University of South Florida, where she served for 15 years as Director of Program Review, leading College and University efforts in accreditation. Her career has been dedicated to standards-based assessment of programs and teachers, beginning with the creation of an evaluation model for accreditation in 1987, which she implemented in several states. Beginning in 1990 for 15 years, she served as the primary consultant for higher education to the Florida Department of Education, where she drafted the standards for the initial approval of teacher education programs, designed the program approval process, and provided technical assistance to colleges of education in evaluation of teachers and programs statewide. She has consulted nationally on NCATE accreditation and worked with state associations of teacher educators on accreditation related issues. She has also consulted with school districts in Florida on assessment systems for teachers. She was lead designer of the assessment system for the Florida Alternative Certification Program, now used in over 40 of the 68 school districts in the State. William Steve Lang is a Professor of Educational Measurement and Research at the University of South Florida St. Petersburg, where he teaches graduate courses in measurement, statistics, and research.   He, too, focuses his teaching on making meaningful and pragmatic uses of the disciplines he teaches.  He earned his Ph.D. from the University of Georgia in 1984.  He has taught as a public school teacher in South Carolina and Georgia and as a college faculty member in South Carolina, Georgia, and Florida.   He has published on a variety of applications in educational testing and works extensively with the Rasch Model of Item Response Theory.  He began working extensively with Judy when she joined the faculty of the St. Petersburg Campus in 2001.  Since that time, they have collaborated in all aspects of their research and service efforts with the Florida Department of Education, Florida school districts, and teacher education programs nationwide.   They are working together to build two teacher assessment scales – one on teacher competencies, the subject of this book, and another on dispositions.  Their work in both areas is standards-driven.

"The text has been well researched, is standards-based, and includes activities, worksheets, definitions, and rubrics. It addresses a topic that has been a mystery to assessment gurus."

• List of TablesList of FiguresList of BoxesList of Abbreviations and AcronymsForeword by Richard C. KunkelPrefaceAcknowledgmentsAbout the Authors1. What Are Dispositions and Why Should We Measure Them?What This Chapter Is AboutThe Importance of Measuring DispositionsThe ChallengeWhat Are Standards-Based Dispositions?Hierarchical Relationships Among Knowledge, Skills, and DispositionsRemembering BloomDispositions and Accreditation Requirements: Requirements and DefinitionsMeasuring Dispositions: Sources of ConfusionMeasuring Dispositions: Morals, Ethics, or Standards Based?Different Construct, Different Assessments, Similar Assessment Design ProcessWrap UpActivity 1.1: Questions for ExplorationActivity 1.2: What Have You Noticed?Activity 1.3: Assessment Belief ScaleActivity 1.4: Cognitive, Affective, and Psychomotor Objectives and Assessments2. Methods for Assessing DispositionsWhat This Chapter Is AboutA Conceptual Framework for Measuring DispositionsMeasuring Teacher Dispositions: The State of the ArtBack to Basics: Bloom and KrathwholAvailable Methods for Measuring Dispositions or AffectThe Importance of Inference in Measuring DispositionsWrap UpActivity 2.1: Questions for ExplorationActivity 2.2: Bloom and the INTASC PrinciplesActivity 2.3: Field WorkActivity 2.4: Review Your Feelings3. DAATS Step 1: Assessment Design InputsWhere We Have Been So FarWhat This Chapter Is AboutWhy Are Purpose, Use, Propositions, and Content So Important?DAATS Step 1A: Define the Purpose(s) and Use(s) of the SystemDAATS Step 1B: Define the Propositions or Principles That Guide the SystemDAATS Step 1C: Define the Conceptual Framework or Content of the SystemDAATS Step 1D: Review Local Factors That Impact the SystemWrap UpWorksheet 3.1: Purpose, Use, Propositions, Content, and Context ChecksheetWorksheet 3.2: Purpose, Use, Content, DraftWorksheet 3.3: PropositionsWorksheet 3.4: Contextual Analysis4. DAATS Step 2: Planning With a Continuing Eye on Valid Assessment DecisionsWhere We Have Been So FarWhat This Chapter Is AboutDAATS Step 2A: Analyze Standards and IndicatorsAll Those IndicatorsWhy Bother?DAATS Step 2B: Visualize the Teacher Demonstrating the Affective TargetsDAATS Step 2C: Select Assessment Methods at Different Levels of InferenceDAATS Step 2D: Build an Assessment Framework Correlating Standards and MethodsWrap UpDAATSWorksheet 4.1: Organizing for Alignment (Version 1)Worksheet 4.2: Organizing for Alignment (Version 2)Worksheet 4.3: Visualizing the Dispositional StatementsWorksheet 4.4: Selecting Assessment Methods for INTASC IndicatorsWorksheet 4.5: Assessment Methods for INTASC Indicators: BlueprintWorksheet 4.6: Cost/Benefit and Coverage Analysis of Assessment Methods5. DAATS Step 3: Instrument DevelopmentWhere We Have Been So FarWhat This Chapter Is AboutDAATS Step 3A: Draft items for Each InstrumentThurstone Agreement ScalesQuestionnaires, Interviews, and Focus GroupsObserved PerformanceThematic Apperception Tests or Situation Reflection AssessmentDAATS Step 3B: Review Items for Applicability to Values, Domain Coverage, Job RelevanceWrap UpWorksheet 5.1: Creating ScalesWorksheet 5.2: Creating Questionnaires, Interviews, or K-12 Focus Group ProtocolsWorksheet 5.3: Creating an Affective Behaviour ChecklistWorksheet 5.4: Creating an Affective Behaviour Rating ScaleWorksheet 5.5: Creating a Tally Sheet for Affective ObservationWorksheet 5.6: Checklist for Reviewing Scale DraftsWorksheet 5.7: Review Sheets for Questionnaires and InterviewsWorksheet 5.8: Review Sheets for K-12 Focus Group ProtocolsWorksheet 5.9: Checklist for Reviewing Observations and Behavioral ChecklistsWorksheet 5.10: Coverage CheckWorksheet 5.11: Rating Form for Stakeholder Review6. DAATS Step 4: Decision Making and Data ManagementWhere We Have Been So FarWhat This Chapter Is AboutDAATS Step 4A: Develop Scoring RubricsDichotomous Response Scoring KeysRating Scale RubricsDAATS Step 4B: Determine How Data Will Be Combined and UsedNeed for Shared DataData StorageData AggregationMaximizing the Utility of the Data for Decision MakingDAATS Step 4C: Develop Implementation Procedures and MaterialsPreponderance of the Evidence vs. Cut ScoresAdvising and Due ProcessScoring ProceduresImplementationWrap UpWorksheet #6.1: Explanation of Dichotomous Scoring DecisionsWorksheet #6.2: Rubric DesignWorksheet #6.3: Sample Format for Candidate/Teacher Tracking FormWorksheet #6.4: Format for Data AggregationWorksheet #6.5: Sample Disposition Event ReportWorksheet #6.6: Management Plan7. DAATS Step 5: Credible DataWhere We Have Been So FarWhat This Chapter Is AboutWhat Is Psychometric Integrity and Why Do We Have to Worry About It?DAATS Step 5A: Create a Plan to Provide Evidence of Validity, Reliability, Fairness, and UtilityElements of a PlanElement 7.1: Purpose and UseElement 7.2: Construct MeasuredElement 7.3: Interpretation and Reporting of ScoresElement 7.4: Assessment Specifications and Content MapElement 7.5: Assessor/Rater Selection and Training ProceduresElement 7.6: Analysis MethodologyElement 7.7: External Review Personnel and MethodologyElement 7.8: Evidence of Validity, Reliability, and Fairness (VRF)Psychometric Evidence Collected AlreadyNext Steps in Collecting Evidence of Validity, Reliability, and FairnessFuture StudiesDAATS Step 5B: Implement the Plan ConscientiouslyWrap UpWorksheets and ExamplesWorksheet 7.1: Assessment SpecificationsWorksheet 7.2: Analysis of Appropriateness of Decisions for Teacher FailuresWorksheet 7.3: Analysis of Rehire DataWorksheet 7.4: Program Improvement RecordWorksheet 7.5: Expert RescoringWorksheet 7.6: Fairness ReviewWorksheet 7.7: Analysis of Remediation Efforts and EO ImpactWorksheet 7.8: Psychometric Plan FormatExample 1: Logistic Ruler for Content ValidityExample 2: Computation of the Lawshe (1975) Content Validity RatioExample 3: Disparate Impact AnalysisExample 4: Computation of Cohen′s Kappa (1960) for Inter-rater ReliabilityExample 5: Two Pearson Correlation Coefficients and Scatterplots: Disposition Scores Correlated with PRAXIS and Portfolio ScoresExample 6: Spearman Correlation Coefficient and Scatterplot: Disposition Scores Correlated With Principal RatingsExample 7: Correlation Matrix and Scatterplots Knowledge, Impact, Dispositions, Skills (KIDS)Example 8: T-Test Comparing Dispositions of Mathematics and Science TeachersExample 9: DIF Analysis for Programs8. Using Teacher Scores for Continuous ImprovementWhat This Chapter Is AboutReasons Why We Use the Rasch ModelThe Classical ApproachA Quick Overview of Where Rasch Fits Into the Grand Scheme of IRT ModelsRasch: The BasicsGetting StartedDifferences That Item Writers MakeGuttman ScalingA Sample Rasch RulerFrom Pictures to NumbersThe Fit StatisticGain Scores " Real or Imagined′Ratings and RatersLearning More About RaschWrap UpActivity #1: Decision-Making Tool for Measurement9. Legal IntegrityWhat This Chapter Is AboutWhy Not Portfolios?Why the Pied Piper?What IF?? A Legal Scenario: Mary Beth JoAnne Sues XYZ UniversityMBJ Helps Us to Understand the Convergency of Psychometrics and Legal RequirementsBackground FactsScenario #1Scenarios #2, 3, and 4Psychometric Issues and Legal Challenges in the Real WorldLegal Issues and PrecedentsThree Landmark Dispositions Cases in Two YearsTide Changing in NCATEStandards Are the Vanguard!MBJ RevisitedEnd NoteResource I. DAATS Steps and WorksheetsResource II. INTASC Disposition IndicatorsGlossaryIndex