Evaluation in Education and Human Services – serie

Scoring Performance Assessments Based on Judgements focuses on the applications of Generalizability Theory to Performance Assessment. The author developed an objective method to examine the dependability of the scoring of large-scale performance assessments by humans. This book presents principles in identifying common missing data patterns inherent in assessment designs. These principles can be used to guide the design of assessment procedures including those used in large-scale testing programs, observations, and structured interviews.

Evaluation Models is an up-to-date revision of the classic text first published in 1983. Organized in three sections, the first includes a historical perspective on the growth of evaluation theory and practice and two comparative analyses of the various alternative perspectives on evaluation. The second section includes articles representing the major schools of thought about evaluation written by the leaders who have developed these schools and models. The final section describes and discusses the Standards for Program Evaluation and the reformation of program evaluation.

We are glad to have the opportunity to work together again in the planning and preparation of this edited volume on the evaluation of corporate training. Our respective professional careers have provided us with experience in this area, both as practitioners and as academicians. It is from both of these perspectives that we approached the preparation of this volume. Our purpose is to provide training professionals in business and industry, and students of human resources development with an overview of current models and issues in educational evaluation. The book is organized around three themes: context, models, and issues. The chapters in the context section are intended to provide the reader with an understanding of the social, organizational, and interpersonal factors that provide background and give meaning to evaluation practice. The models section brings together contributions from some of the most influential thinkers and practitioners in the field. The chapters in this section provide perspective on the dominant themes and emergent trends from individuals who have been, and continue to be, the drivers of those trends. Contributions to the issues section highlight some pervasive themes as well as illuminate new areas of concern and interest that will affect how we assess learning interventions in the organizations of today and tomorrow.

Constructing test items for standardized tests of achievement, ability, and aptitude is a task of considerable importance. The interpretability of a test's scores flows directly from the quality of its items and exercises. Concomitant with score interpretability is the notion that including only carefully crafted items on a test is the primary method by which the skilled test developer reduces unwanted error variance, or errors of measurement, and thereby increases a test score's reliability. The aim of this text is to increase the test constructor's awareness of this source of measurement error, and then to describe methods for identifying and minimizing it during item construction and later review. Persons involved in assessment are keenly aware of the increased attention given to alternative formats for test items in recent years. Yet, the text argues that in many writers' zeal to be "curriculum-relevant" or "authentic" or "realistic", the items are often developed seemingly without conscious thought to the interpretations that may be garnered from them.It also argues that the format for such alternative items and exercises also requires rigour in their construction and even offers some solutions, as one chapter is devoted to these alternative formats. The book addresses issues in constructing test items by focusing on four ideas. First, it describes the characteristics and functions of test items. A second feature of this book is the presentation of editorial guidelines for writing test items in all of the commonly used item formats, including constructed-response formats and performance tests. A third aspect is the presentation of methods for determining the quality of test items. Finally, it presents a compendium of important issues about test items, including procedures for ordering items in a test, ethical and legal concerns over using copyrighted test items, item scoring schemes, computer-generated items and more.

This text revisits vital issues of equity and assessment through the research efforts and insights of many of America's most prominent educators and assessment experts. As its most urgent purpose, the publication aims to sensitize readers to the unfairness and inappropriate uses of testing instruments which under optimal circumstances have the potential to benefit all students. With America fervently espousing both national and state testing, the differential performance by race and social class raises the spectre of tests as barriers to life milestones such as promotion, graduation, and college admissions. In response to such punitive testing, the papers included here explore a host of models and practices that are being piloted both in America and other countries as educators grapple with the effects the assessment is having on minority and disadvantaged students and school systems. In the process, outcomes of innovative portfolio and authentic assessments are weighed against important standards and principles of validity and consequences.As the various authors probe the gap between African-American and white test scores, they raise important questions of resources, family background and educational opportunity. Beyond the value of their recommendations to educators, their papers help to identify causes of pupil deficiencies in ways that can be addressed by policymakers. To reinforce the emphasis on equity, several authors present a definitive defense of affirmative action as a critical counter-measure to the lack of fairness in school quality, family and social supports, and educational resources.

Bernard R. Gifford In the United States, the standardized test has become one of the major sources of information for reducing uncertainty in the determination of individual merit and in the allocation of merit-based educational, training, and employment opportunities. Most major institutions of higher education require applicants to supplement their records of academic achievements with scores on standardized tests. Similarly, in the workplace, as a condition of employment or assignment to training programs, more and more employers are requiring prospective employees to sit for standardized tests. In short, with increasing frequency and intensity, individual members of the political economy are required to transmit to the opportunity marketplace scores on standardized examinations that purport to be objective measures of their abilities, talents, and potential. In many instances, these test scores are the only signals about their skills that job applicants are permitted to send to prospective employers. THE NATIONAL COMMISSION ON TESTING AND PUBLIC POLICY In view of the importance of these issues to our current national agenda, it was proposed that the Human Rights and Governance and the Education and Culture Programs of the Ford Foundation support the establishment of a ''blue ribbon" National Commission on Testing and Public Policy to investigate some of the major problems, as well as the untapped opportunities, created by recent trends in the use of standardized tests, particularly in the workplace and in schools.

Bernard R. Gifford In the United States, the standardized test has become one of the major sources of information for reducing uncertainty in the determination of individual merit and in the allocation of merit-based educational, training, and employment opportunities. Most major institutions of higher education require applicants to supplement their records of academic achievements with scores on standardized tests. Similarly, in the workplace, as a condition of employment or assignment to training programs, more and more employers are requiring prospective employees to sit for standardized tests. In short, with increasing frequency and intensity, individual members of the political economy are required to transmit to the opportunity marketplace scores on standardized examinations that purport to be objective measures of their and potential. In many instances, these test scores are the abilities, talents, only signals about their skills that job applicants are permitted to send to prospective employers. THE NATIONAL COMMISSION ON TESTING AND PUBLIC POLICY In view of the importance of these issues to our current national agenda, it was proposed that the Human Rights and Governance and the Education and Culture Programs of the Ford Foundation support the establishment of a ''blue ribbon" National Commission on Testing and Public Policy to investigate some of the major problems as well as the untapped opportunities created by recent trends in the use of standardized tests, particularly in the workplace and in schools.

In 1976, the first session on the teaching of evaluation was held at an annual meeting of evaluators. A few hardy souls gathered to exchange ideas on improving the teaching of evaluation. At subsequent annual meetings, these informal sessions attracted more and more participants, eager to talk about common teaching interests and to exchange reading lists, syllabuses, assignments, and paper topics. The ses­ sions were irreverent, innovative, lively, and unpredictable. Eventually the group for­ malized itself with the American Evaluation Association as the Topical Interest Group in the Teaching of Evaluation (TIG: TOE). As word of TIG: TOE's activities spread, instructors from all over the country clamored for assistance and advice. It became apparent that a handbook was need­ ed, a practical interdisciplinary guide to the teaching of evaluation. Donna M. Mertens, a long-standing member of TIG: TOE and an accomplished teacher of evaluation, volunteered to edit the book, and her skills, sensitivity, and experience in the craft of teaching are apparent throughout.

My interest in and appreciation for program evaluation began in the early 1970's when conducting a curriculum development research project at the University of Florida's P. K. Y onge Laboratory School. This interest was sparked when it became apparent that testing the success of an education program required more skills than just statistics and research methods. After pursuing additional formal schooling, I embarked on a career featuring educational program evaluation as its central thrust--as a private consultant, later in a university health sciences center involving seven academic colleges, and then in the Cooperative Extension Services of Florida and Maryland. Adding evaluability assessment (EA) to the performance of evaluations, to program development, and to teaching about evaluation has been a significant development for me personally, and I hope to those who have been participants with me in each endeavor. This book grew out of many of these experiences and involved numerous colleagues who made significant contributions. First among these is Dr. George Mayeske, Program Evaluation Specialist, Extension Service, U. S. Department of Agriculture, Washington, D. c.

J. T. Sandefur Western Kentucky University American's ability to compete in world markets is eroding. The productivity growth of our competitors outdistances our own. The capacity of our economy to provide a high standard of living for all our people is increasingly in doubt. As jobs requiring little skill are automated or go offshore and demand increases for the highly skilled, the pool of educated and skilled people grows smaller and the backwater of the unemployable rises. Large numbers of American children are in limbo--ignorant of the past and unprepared for the future. Many are dropping out--notjust out of school--but out of productive society. These are not my words. They are a direct quote from the Executive Summary of the Carnegie Forum Report on Education and the Economy entitled A Nation Prepared: Teachers for the 21st Century (p. 2, 1986). This report was motivated by four purposes: 1. To remind Americans, yet again, of the economic challenges pressing us on all sides; 2. To assert the primacy of education as the foundation of economic growth, equal opportunity and a shared national vision; 3. To reaffirm that the teaching profession is the best hope for establishing new standards of excellence as the hallmark of American education; and 4. To point out that a remarkable window of opportunity lies before us in the next decade to reform education, an opportunity that may not present itself again until well into the next century.

Over the last 20 years there have been a large number of technical advances and changes in the field of educational and psychological testing. According to Anne Anastasi, The decade of the 1980's has been a period of unusual advances in ,psychological testing. Technological progress, theoretical sophistication, and increasing pro­ fessional responsibility are all evident in the fast-moving events in this field (A. Anastasi, Psychological Testing, Sixth Edition. New York: Macmillan, 1988). On the psychometric front, advances in topics such as item response theory, criterion-referenced measurement, generalizability theory,· analy­ sis of covariance structures, and validity generalization are reshaping the ways that ability and achievement tests are constructed and evaluated, and that test scores are interpreted. But \Jsychometric advances, as substantial and important as they have been, are only a fraction of the major changes in the field of testing. Today, for example, the computer is radically chang­ ing the ways in which tests are constructed, administered, and scored. Computers are being used to administer tests "adaptively." That is, the sequence of questions an examinee is administered depends upon his or her performance on earlier administered items in the test. Tests are "adapted" to the ability levels of the examinees who are being assessed. One result is shorter tests with little or no loss in measurement precision. Computers are also being used to store or bank test items. Later, items of interest can be selected, and the computer is used to print copies of the test.

Bernard R. Gifford and Linda C. Wing Standardized testing has become a ubiquitous feature of American life. As a major source of information for reducing uncertainty in the alJocation of merit­ based educational, training, and employment opportunities, testing affects the life chances of individuals. Moreover, testing inOuences the way in which our societyjudgesitselfandprovides for ourcollective future. Test scores may determine a child's admission to lcindergarten and promotion to the fIrst grade. Most states award the high school diploma only ifa student has passed a minimum competency test. Major institutions of higher education typically require applicants to supplement their records of academic achievement with scores on college admissions tests. In the labor market, as a condition of employment or assignment to training programs, more and more employers are requiring workers to sit for personnel selection tests. Additionally, it has become commonplace to use test scores to calibrate our national sociopolitical condition and our capacity to compete with other countries in the global economy. In short, with increasing frequency and intensity, scores on examinations that purport to be objective and precise measures of individual knowledge, abilities, and potential are playing a critical role in the opportunity marketplace. Similarly, test scores are exercising growing influence in assessments of our social and economic institutions and in policy decisions about the relative invesunents that should be made in each. In all these instantiations, test scores are at the center of high-stakes decision making about the future of individuals and of the nation itself.

Bernard R. Gifford As we edge toward the year 2000, the information age is a reality; the global marketplace is increasingly competitive; and the U.S. labor force is shrinking. Today more than ever, our nation's economic and social well-being hinges on our ability to tap our human resources-to identify talent, to nurture it, and to assess abilities and disabilities in ways that help every individual reach his or her full potential. In pursuing that goal, decision-makers in education, industry, and government are relying increasingly on standardized tests: sets of question- with identical directions, time limits and tasks for all test-takers-designed to permit an inference about what someone knows or can do in a particular area. CALIBRATING DIFFERENCE Our emphasis on standardized testing rests on a premise that is so basic it often escapes notice: that we humans are different from each other in ways that are both meaningful and measurable. We differ in terms of cognitive ability; aptitude for performing different kinds of mental and physical tasks; temperament; and interests. But somehow, without sufficient examination, we have taken a great collective leap from that commonplace to the notion that there are precise, measurable gradations of innate ability that can be used to direct children to the right classrooms, and adults to the right job slots.

Deals with various issues within the field of educational assessment, emphasizing those which have sparked the US public policy debate in recent years. Much of this volume centres on the impact of testing on various subgroups of the population, including blacks, hispanics and young children.

Details a unique training evaluation approach developed by David J. Basarab, formerly Manager of Educational Measurement at NCR and currently Manager of Evaluation at Motorola University, which has enabled these corporations to successfully evaluate their corporate training programmes. This allows corporations to determine whether money spent on training is an investment or an expense. This evaluation approach is alos significant in determining either the effectiveness of or the opportunities to improve corporate training. In this book, the authors provide the commercial training industry with a step-by-step approach to use when evaluating training programmes. It can be applied generically to evaluate traning programmes across all corporate environments.

Standardized testing in the United States has been increasing at a rapid pace in the last twenty-five years. The market for tests has not only been expanding rapidly, but has also been changing sharply in structure into a fractured marketplace. Indeed, one of the main features of this book is that the market for standardized testing is highly fractured - with segments of the market facing monopoly conditions, others facing oligopoly conditions and still others where near free-market conditions exist.One of the main premises of the book is that the structures of markets have strong implications for how those markets perform. While this notion is widely accepted among economists, it is not widely appreciated in educational research. A second motivation for the book is that very little scholarly attention has been focused on the standardized testing industry. This topic - the structure of the testing industry and implications for the quality of tests and test use - affects how we evaluate the learning of students, the effectiveness of teaching, the quality of schools and the educational health of the nation. Of particular concern to the authors is one vital aspect of test quality: test validity.This book is the most current and authoritative review and analysis of the market for standardized testing.

In response to the emerging need to develop teachers as professionals who evaluate their own work, this book presents the foundations of self-evaluation as well as self-evaluation models and tools that are likely to help educational practitioners to evaluate their own teaching, and thus raise the level of their professional functioning.The book is intended to serve several groups: student teachers whose socialization into the teaching profession should include the perception of self-evaluation as an inherent part of teaching; the student teachers' supervisors who are expected to help in developing the knowledge and skills that are needed for purposes of self-evaluation; and teachers, school principals, and university instructors in departments of teacher education, who are interested in teacher's growth and in the development of teaching as a profession.

Drawing upon experiences at state- and local-level project evaluation, and based on current research in the professional literature, Payne presents a practical, systematic and flexible approach to educational evaluations. Evaluators at all levels - state, local and classroom - should find ideas useful in conducting, managing and using evaluations. Special user targets identified are: state department of education personnel and local school system administrative personnel. The volume can be used by those doing evaluation projects "in the field", or as a text for graduate courses at an introductory level. The book begins with an overview of the generic evaluation process. Chapter Two is devoted to the criteria for judging the effectiveness of evaluation practice. Chapter Three addresses the topic of evaluation goals and objectives. Chapters Four, Five and Six are concerned with the approach, framework, or design of an evaluation study. Chapter Four contains a discussion of four major philosophical frameworks or metaphors and the implications of these frameworks for conducting an evaluation.Chapters Five and Six describe predominantly quantitative and qualitative designs, respectively. Design, implementation and operational issues related to instrumentation (Chapter Seven), management and decision making (Chapter Eight), and reporting and utilization of results (Chapter Nine) are next addressed. The final chapter of the book (Chapter Ten) considers the evaluation of educational products and materials.

The goals and content of this text are derived from three ongoing efforts: to advance the institution of education and to promote educational opportunities to children and youth worldwide; to promote effective assessment policies and practices that enhance sound educational practice; and to address the need to develop tests and other assessment practices in less developed countries as well as to augment and alter a number of traditional assessment practices in developed nations.

This text focuses on the following four issues: the growing prominence that educational testing and assessment are playing in American education; the effect that current testing and assessment policies and practices are having upon minority and disadvantaged students and school systems; the new and innovative approaches to testing and assessment and their prospects for promoting educational equity, educational reform and accountability; and the actions that need to be taken in order to achieve equity in testing and assessment in the nation's schools. The goal is to identify and begin addressing some of the disadvantages faced by many Americans in their pursuit of high-quality education and training. It explores some of the factors that contribute to the gaps in student achievement along racial and social class lines, and offers ideas for improving the measurement of America's diverse population of students. Some of the chapters focus on how the field of educational testing is changing to include new types of assessment practices and policies, and new types of testing and assessment instruments.Innovations in educational testing and assessment include student exhibitions, portfolios, performance tasks, and essay writing, all of which are being advanced both for in-classroom and in-school use as well as for purposes of public accountability. Some of the public policy innovations include the following: awarding federal funding and recognition based upon student performance on tests and assessments; national government approval of the assessments that states, school districts, and schools use; and national approval and sanctioning of the standards that will be measured by educational tests and assessments.

This volume is organized around four dominant, interrelated core issues: professional standards; a guide to applying the Joint Committee's Standards; ten alternative models for the evaluation of teacher performance; and an analysis of these selected models. The book draws on research and development conducted by the federally-funded national Center for Research on Educational Accountability and Teacher Evaluation (CREATE). It aims to explain the essence of sound teacher evaluation in such a way that readers will be able to apply its principles, facts, ideas, processes, and procedures. In addition, it explores the latest developments in teacher evaluation.

Corporate training and effective performance have become major issues in the 1980s and '90s. Reviews of the training research literature show that, parallel to the growing attention to corporate training, research has also increased in the field, giving a better understanding of the subject and providing fundamental expertise on which trainers can build. The contributions to the book differ in the degree to which they are related to performance issues, but all chapters underline the necessity of thinking from the perspective of effective performance.

Achievement assessment has undergone a major shift, from what some call a 'culture of testing' to a 'culture of assessment'. Nowadays, a strong emphasis is placed on the integration of assessment and instruction, on assessing processes rather than just products, and on evaluating individual progress relative to each student's starting point. This book addresses assessment issues in light of the present state of affairs. The first part discusses new alternatives in the assessment of achievement in various subject areas, focusing on agenda, practice, impact and evaluation of the assessment. The second part deals with issues related to assessment of the learning process, specifically: questions concerning the assessment of individual differences in prior knowledge, learning skills and strategies.

This text is organized around four dominant, interrelated core issues: professional standards; a guide to applying the Joint Committee's Standards; ten alternative models for the evaluation of teacher performance; and an analysis of these selected models. The book draws heavily on research and development conducted by the Federally-funded National Center for Research on Educational Accountability and Teacher Evaluation (CREATE). It aims to teach the essence of sound teacher evaluation and help readers to apply its principles, facts, ideas, processes and procedures. Finally, the book invites and assists school professionals and other readers to examine the latest developments in teacher evaluation.

Every school district needs a system of sound superintendent performance evaluation. School district superintendents are and must be accountable to their school boards, communities, faculties, and students for delivering effective educational leadership. To assure that they are evaluated fairly, competently, and functionally, superintendents need to help their school boards plan and implement evaluation systems that adhere to the evaluation standards. This text outlines some of the problems and deficiencies in current evaluation practice and offers professionally-based leads for strengthening or replacing superintendent performance evaluation systems. The focus is on the on-the-job performance of school district superintendents as they implement school board policy. The decision to focus on performance evaluation reflects the importance of this kind of evaluation in the move to raise educational standards and improve educational accountability. Boards and superintendents are advised to make superintendent performance evaluation an integral part of the district's larger system for evaluating district needs, plans, processes, and accomplishments.

Please glance over the questions that follow and read the answers to those that are of interest. Q: What does this manual do? A: This manual guides the user through designing an evaluation. A: Who can use it? A: Anyone interested or involved in evaluating professional trammg or inservice education programs. The primary users will be staff members who are doing their own program evaluation-maybe for the first time. (Experienced evaluators or other professional educators can find useful guides and worksheets in it.) Q: If I work through this manual, what will I accomplish? A: You will develop one or more evaluation designs, and perhaps you'll also use the designs to evaluate something to make it better or to document its current value. Q: What is an evaluation design? A: An evaluation design is a conceptual and procedural map for getting important information about training efforts to people who can use it, as shown in the graphic below.

Phil R. Manning "Can you prove that continuing education really makes any difference?" Over the years, educators concerned with continuing education (CE) for health professionals have either heard or voiced that question in one form or another more than once. But because of the difficulty in measuring the specific effects of a given course, program, or conference, the question has not been answered satisfactorily. Since CE is costly, since CE is now mandated in some states for re-registration, and since its worth has not been proven in for­ mal evaluation research, the pressure to evaluate remains strong. The question can be partially answered by a more careful definition of continuing education, particularly the goals to be achieved by CEo Another part of the answer depends on the development of a stronger commitment to evaluation of CE by its providers. But a significant part of the answer might be provided through the improvement of methods used in evaluation of continuing education for health professionals. To address this last concern, the Development and Demonstration Center in Continuing Education for the Health Professions of the Univer­ sity of Southern California organized and conducted a meeting of academi­ cians and practitioners in evaluation of continuing education. During a three-day period, participants heard formal presentations by five invited speakers and then discussed the application of the state of the art of educa­ tional evaluation to problems of evaluation of continuing education for health professionals.