The preliminary experiences of a process evaluator contracted to evaluate a three year program to develop Alaskan Native leadership are presented. With hindsight, some characteristics of this particular project having implications for a process evaluator are noted and some general comments on process evaluation, based on the author's experiences, are generated. (RC)
Topics: ERIC Archive, Evaluation Methods, Formative Evaluation, Professional Personnel, Program Evaluation
The intent of formative evaluation is to improve programs as well as to justify their continuation. It is critical to separate clearly those functions of the evaluation which are political from those which may lead the way to instructional improvement. Data for formative evaluation should be gathered in an interpretable way at the level at which decisions will be implemented, usually at the classroom level. This forces data collection activities to a single planned incursion. This...
Topics: ERIC Archive, Data Collection, Evaluation Methods, Formative Evaluation, Instructional Improvement
Though instructional development models stress the importance of formative evaluation, techniques for the formal evaluation of video tape programs remain primitive. The attention, attitude, and adoption levels achieved can be assessed by use of the following evaluation instruments: (1) expert appraisal of script; (2) special questionnaires; (3) story board; (4) content analysis; (5) stop frame; (6) observation--personal or video; (7) eye movement; (8) demand for audio-video; (9) real time...
Topics: ERIC Archive, Educational Television, Evaluation Methods, Evaluation Needs, Formative Evaluation,...
It is assumed that the delineation of values is critical to proper needs assessments. "Need" is defined as a requisite or desideratum generated as a discrepancy between a target state (T) and an actual state (A), if and only if the presence of the conditions defined by T can be shown significantly to harm, indispose, or constrain a subject (S). There are at least five points at which value judgments impinge upon needs determination. They are: (1) the identification of the domain of T;...
Topics: ERIC Archive, Educational Assessment, Evaluation Methods, Formative Evaluation, Needs Assessment,...
The effects of two data-utilization rules on spelling achievement were compared for an 11-year-old boy who had been diagnosed as learning disabled (LD). During instructional sessions, the boy was taught and measured on sets of difficult spelling words. Graphed data were analyzed using a concurrent schedule design whereby equivalent behaviors are treated simultaneously with different approaches to determine relative treatment effects. One treatment approach involved the following...
Topics: ERIC Archive, Elementary Education, Formative Evaluation, Learning Disabilities, Spelling, Student...
This document reports on how formative evaluation was used by the National Institute for Science Education (NISE) to achieve its stated goals and to develop an institute from an amorphous group. It was found that formative evaluation should provide ideas, information, models, and language that an emerging interdisciplinary organization can use to reframe and revise what it is accomplishing and what kind of organization it is becoming. It should enhance the leaders' capacity to function as...
Topics: ERIC Archive, Formative Evaluation, Interdisciplinary Approach, Organizational Theories,...
For the last three years the Military Training Centers (MTC) PLATO support group has been experimenting with the development of a number of formalized techniques for assisting authors in writing instructionally effective courseware. These methods, collectively referred to as "lesson reviewing," vary widely depending on individual authors' needs. Basically, a lesson review is a set of comments about the lesson ranging from alternative instructional strategy suggestions to grammatical...
Topics: ERIC Archive, Authors, Feedback, Formative Evaluation, Instructional Materials, Review...
Although there are fundamental differences in the objectives of the two activities, the programing of instructional materials bears many similarities to the construction of tests. A systematic comparison of the problems and procedures reveals important implications for programing from the older field of testing. Theory and experience in test construction can be especially useful in the selection of valid criteria for assessing the effectiveness of a program, the ordering of instructional...
Topics: ERIC Archive, Formative Evaluation, Instructional Design, Material Development, Programed...
This issues paper, seventh in a series of eight, is intended to distill formative evaluation questions on topics that are central to the development of the higher and further education information environment in the United Kingdom. The aim of this issues paper is to provide a description of the approach of the Formative Evaluation of the Distributed National Electronic Resource (EDNER) Project to understanding implicit theories of change to allow project teams (learning resource development...
Topics: ERIC Archive, Change, Electronic Classrooms, Foreign Countries, Formative Evaluation, Higher...
Online assessment of any type is attractive to educators because it allows large amounts of student data to be gathered and accessed quickly. Indeed, online, summative assessments--generally end-of-course, state-level, and placement exams--are making the annual student testing ritual a meaningful audit instead of what some call an "autopsy" because data reports arrive too late to have any impact on student learning. Yet educators now want meaningful data sooner and more regularly to...
Topics: ERIC Archive, Standards, Computer Assisted Testing, Conferences, Educational Testing, Formative...
The Next Generation Assessment Task Force was convened to formulate Wisconsin's path forward. Task force members listened to leaders from business and technology sectors as well as leaders from PK-12 and higher education. This summary shares the process, definitions, assumptions, and recommendations of the task force. This paper aims to use these findings as a blueprint for the next generation of assessment. [This report was supported by the Great Lakes West Comprehensive Assistance Center.]
Topics: ERIC Archive, Student Evaluation, Measurement, Academic Achievement, Achievement Tests, Summative...
Although much recent attention has focused on gaps in the achievement of different groups of students, the problem has been with us for decades. This paper presents the problem as one of reducing variation in students' achievement, and reviews the work of renowned educator Benjamin Bloom on this problem. Bloom argued that to reduce variation in students' achievement and to have all students learn well, we must increase variation in instructional approaches and learning time. The key element in...
Topics: ERIC Archive, Educational Researchers, Educational Research, Formative Evaluation, Mastery...
This issues paper, one of a series of eight, is intended to distill formative evaluation questions on topics that are central to the development of the higher and further education information environment in the United Kingdom. The topic of this first issues paper is a conceptual framework that can help members of a project (information resource) development team discuss and clarify key issues about how the resource on which they are working is intended to help students learn. This paper offers...
Topics: ERIC Archive, Cognitive Psychology, College Students, Formative Evaluation, Higher Education,...
The paper examines the need for improvement in accountability and evaluation in rehabilitation programs for handicapped persons, with an emphasis on formative, or internal evaluation. Three factors significantly related to program management and accountability are explored: a client's need status, staff functions and attitudes, and outcome studies. A procedure for determining a client's need status on the basis of resource allocation (staff position and time) is offered. Possible staff...
Topics: ERIC Archive, Accountability, Cost Effectiveness, Disabilities, Evaluation Methods, Formative...
Results are reported of research to test whether a simulation game can be used to identify and to influence the leadership styles practiced by elementary school principals. The design assesses whether principals can become more skillful decisionmakers through heightened awareness of their leadership styles. The research and educational implications of the study contribute to "middle range" theorizing. Application lies between general managerial and systems theory, and the day-to-day...
Topics: ERIC Archive, Decision Making, Educational Games, Elementary Education, Formative Evaluation,...
As a part of a project to develop formative evaluation systems for teachers to improve learning disabilities (LD) service programs in written expression, three concurrent validity studies were conducted on the relationship between performances on behavioral measures of written expression and performances on standardized achievement measures of written expression. Results of correlational analyses are reported for five behavioral measures, two standardized measures, and the Developmental...
Topics: ERIC Archive, Academic Achievement, Evaluation Methods, Exceptional Child Research, Formative...
This paper examines the concepts of program monitoring and program evaluation in the literature, and offers working definitions based on two dimensions of measurement: focus (what questions are addressed) and timing (how often the measures are taken). Focus can be on inputs to the program or outcomes from it; timing can be one-shot or continuous. Continuous measures are here considered to be monitoring--"input monitoring" if the measures are on inputs, "impact monitoring" if...
Topics: ERIC Archive, Evaluation Methods, Evaluators, Formative Evaluation, Program Administration, Program...
Conventional wisdom says that a person can never plan enough or be too prepared. Conventional wisdom is often correct. Many successful schools--those that ensure that all students achieve at high levels--follow a detailed and comprehensive school improvement plan like a blueprint. But if this is so, why is it that many other schools that produce equally detailed and comprehensive plans do not achieve these same results? In trying to deal with the complexities of school improvement, schools...
Topics: ERIC Archive, Educational Planning, Educational Improvement, Needs Assessment, Administrative...
An applied model is presented for the utilization of formative evaluation in developing instructional materials. The model is introduced through a presentation of its conceptual rationale. A description is given of its methodology and instrumentation, and a demonstration presented of its development and utilization within the framework of a project for the development of instructional materials for culturally disadvantaged students at the junior high school level. Practical recommendations are...
Topics: ERIC Archive, Curriculum Development, Decision Making, Evaluation Methods, Formative Evaluation,...
A revision of an earlier rough draft entitled "Local School and System-Wide Planning," this report outlines the planning process. Flow charts and a narrative description of a Central Office and Local School Planning Cycle are provided. This cycle consists of the following steps: (1) superintendent's statement on the implementation of plan cycle, (2) identification of local/department planning team, (3) local/department needs assessment, (4) comparison of local and central office needs...
Topics: ERIC Archive, Educational Planning, Evaluation Methods, Formative Evaluation, Guides, Program...
The current study aims to contribute to classroom instruction of undergraduate education faculty students through development of an online contest, and evaluation of test items based on participants' responses. It is a pilot study which covers four units of a single course offered in the Department of Computer Education and Instructional Technologies at Anadolu University. Students are provided with the opportunity to practice their skills on-line whenever and wherever they want. The research...
Topics: ERIC Archive, Computer Assisted Testing, Formative Evaluation, Undergraduate Students, Foreign...
In the first part of this paper we discuss the feasibility of using Generalizability (G) Theory to examine the dependability of concept map assessments and to design a concept map assessment for a particular practical application. In the second part, we apply G theory to compare the technical qualities of two frequently used mapping techniques: construct-a-map with created linking phrases (C) and construct-a-map with selected linking phrases (S). We explore some measurement facets that...
Topics: ERIC Archive, Formative Evaluation, Generalizability Theory, Concept Mapping, Comparative Analysis,...
The Center for Comprehensive School Reform and Improvement's January 2007 newsletter, "A Teacher's Guide to Differentiating Instruction," [ED495740] summarized the following four planning stages teachers can use to effectively differentiate instruction: (1) A thorough understanding of content; (2) Use of preassessments and regular formative assessments; (3) Use of curricular materials and instructional strategies that meet the diverse needs of students; and (4) Use of summative...
Topics: ERIC Archive, Individualized Instruction, Educational Strategies, Newsletters, Formative...
The basic issues, problems, techniques, and history of program evaluation are reviewed, with emphasis on the evaluation of learning assistance programs. The following topics are discussed: (1) historical perspectives (origins of the evaluation process, program evaluation in education); and (2) theoretical constructs (typologies of evaluation designs, Boylan's four-tiered program evaluation model, Clowes's stage model for program evaluation, and a comparative critique of Boyland and Clowes). A...
Topics: ERIC Archive, Evaluation Methods, Formative Evaluation, Higher Education, Models, Program...
This report of a formative evaluation of S.T.E.P. provides a description of the data gathering instruments for this multi-discipline/multi-audience project, an explanation of the multi-attribute utility measurement format, and a discussion of the implementation procedures used. Data were collected by questionnaires, interviews, and anecodotal notes. Examples of these questionnaires are given in appendix A, while comments and recommendations, objectives, and results of the utility assessment of...
Topics: ERIC Archive, Communications Satellites, Evaluation Criteria, Formative Evaluation, Interviews,...
In the current age of accountability, teachers must be able to produce tangible evidence of students' concept mastery. This article focuses on implementation of formative assessments before, during, and after instruction in order to maximize teachers' ability to effectively monitor student achievement. Suggested strategies are included to help expand educators' repertoire of possible formative assessments that can be utilized in the classroom setting. These strategies can make teachers more...
Topics: ERIC Archive, Formative Evaluation, Evaluation Methods, Student Evaluation, Academic Achievement,...
Current practices and controversial issues pertaining to student evaluation of instruction are briefly reviewed and followed by a discussion of the application of formative evaluation to this endeavor. Two reasons are generally given for conducting teacher evaluations by students: identification of good teachers and instructional improvement. Summative evaluation appears to provide the identification but does not appear to improve instruction. The use of summative evaluation alone ignores the...
Topics: ERIC Archive, Elementary Secondary Education, Feedback, Formative Evaluation, Student Evaluation of...
This document presents a model and description of a systematic institutional performance appraisal instrument. Particular emphasis of the instrument is focused on the strengths and weaknesses of the structure that supports the university's operation toward goal achievement. The instrument is intended to be elementary because it is believed that no sophisticated performance evaluation system installed without systematic internal self-evaluative experience would work. (HS)
Topics: ERIC Archive, Educational Research, Evaluation Methods, Formative Evaluation, Higher Education,...
Formative assessment, or assessment for learning, has been championed by assessment specialists and increasingly endorsed by professional organizations. Although it may be particularly beneficial for secondary students, the implementation of formative assessment at the secondary level has met with some resistance. To take stock of the research that has been done in this area, and to consider prospects for further inquiry, a methodical review of recent research was undertaken. Following a...
Topics: ERIC Archive, Formative Evaluation, Secondary Education, Evaluation Research, Literature Reviews,...
Assessment forms an important part of instruction. Assessment that aims to support learning is known as formative assessment and it contributes student's learning gain and motivation. However, teachers rarely use assessment formatively to aid their students' learning. Thus, reviewing the factors that limit or support teachers' practices of formative assessment will be crucial for guiding educators to support prospective teachers in using formative assessment and also eliminate limiting factors...
Topics: ERIC Archive, Formative Evaluation, Student Evaluation, Adoption (Ideas), Influences, Models,...
Two representative approaches to measuring and assessing organizational learning are compared. Based on the benefits and drawbacks of each, an alternative framework is proposed for an assessment based on action science, from pragmatic point of view of members of a team within an organization which has declared an intention to improve learning processes. (Contains 1 table, 1 endnote, and a bibliography.)
Topics: ERIC Archive, Learning Processes, Organizational Development, Questionnaires, Alternative...
This report details the activities of the Education Commission of the States (ECS) during 1999-2000. The accomplishments of the ECS are presented in six parts: constituent/state services; information services; conferences, workshops, and other meetings; organizational performance; key issues; and priorities for 2000 and beyond. The highlights of the agency's work include their provision of expert consultation, policy information, and technical assistance to education policy leaders; the...
Topics: ERIC Archive, Annual Reports, Educational Improvement, Educational Quality, Elementary Secondary...
This appendix to the Mathematical Problem Solving Project "Module Development and Formative Evaluation" contains quizzes for trials 1 and 2 for the module entitled "Using Guesses to Solve Problems." Teachers' comments from these trials are included. (MP)
Topics: ERIC Archive, Elementary Education, Elementary School Mathematics, Formative Evaluation,...
Three types of evaluation used in program evaluation and the type of instrument or measure used are discussed. The three types of evaluation are: comparison group evaluation, analysis of learner outcome or product objectives, and process evaluation. Measurements employed are either direct or indirect. Most measurement in education is usually indirect, by which the subject may be measured while he either actively or passively participates in the measurement act. When it is determined that a test...
Topics: ERIC Archive, Criterion Referenced Tests, Evaluation Methods, Formative Evaluation, Guides,...
In the context of instructional development projects, situations arise in which evaluations are required that are neither formative nor summative in role. Typical responses to such situations are to stretch the meaning of one or the other of the terms introduced by Scriven (1967) to accommodate the situation; a consequence is a loss in precision of language which often results in difficulties in communication among practitioners. Another role descriptor, confirmative evaluation, describes a...
Topics: ERIC Archive, Communication Problems, Decision Making, Evaluation, Formative Evaluation,...
This Digest addresses the benefits of formative assessment and provides examples and resources to support its implementation. Formative assessment is the diagnostic use of assessment to provide feedback to teachers and students over the course of instruction. Since the goal of formative assessment is to gain an understanding of what students know and dont know in order to make responsive changes in teaching and learning, techniques such as teacher observation and classroom discussion have an...
Topics: ERIC Archive, Educational Assessment, Elementary Secondary Education, Evaluation Utilization,...
The data compression aspect of 20th century science is evident in the "mass action" concept of brain function embodied in the "g" model of intelligence and academic tests. This paper sketches the history of the devolution of the "g" model into multifactored intelligences, contrasts dynamic process versus static product assessment, and discusses the effects of process assessment on measurement theory. An alternative neurocognitive processing assessment, as opposed...
Topics: ERIC Archive, Diagnostic Tests, Evaluation Methods, Formative Evaluation, Intelligence,...
This report presents the findings and conclusions, and recommendations for the Formative Evaluation of the Targeted Initiative for Older Workers (TIOW). The TIOW was introduced in 2006 to help older workers in vulnerable communities who had lost their jobs to extend their labour market participation and reintegrate into employment. The TIOW is cost-shared between the Government of Canada and participating provinces and territories. During the evaluation period, nine provinces and territories...
Topics: ERIC Archive, Foreign Countries, Older Workers, Labor Market, Employment Opportunities, Needs...
Described in this process evaluation are the first-year efforts to provide a specialized staff within the Bexar County District Attorney's Office, San Antonio, Texas, to perform advocacy functions for abused and neglected children involved in cases going through the criminal justice system. The report discusses how well the staff met the three objectives of the project: (1) to provide supportive services to alleged victims of child abuse and their families; (2) to develop a system of...
Topics: ERIC Archive, Advocacy, Child Abuse, Child Neglect, Formative Evaluation, Networks, Recordkeeping,...
The formative classroom assessment using cooperative groups described in this paper has four purposes: (1) to increase students' understanding of concepts through verbal interaction with peers; (2) to provide feedback to the instructor on the cognitive processes students use to answer questions; (3) to reinforce the classroom learning environment; and (4) to model a variety of assessment methods. The assessment is based on the work of L. Vygotsky, among others. Students are randomly assigned to...
Topics: ERIC Archive, Cooperative Learning, Feedback, Formative Evaluation, Higher Education, Peer...
This guide provides evaluators interested in new methods with a framework for determining the feasibility, utility, and effectiveness of the methods in field settings. To accomplish this intent, the body of this guide contains discussions of five checklists. Three of the checklists are to be used before a trial to determine the feasibility of one or more methods, and during and after a trial to determine the method's utility and effectiveness. An additional checklist lists factors related to...
Topics: ERIC Archive, Check Lists, Evaluation Methods, Feasibility Studies, Field Studies, Field Tests,...
Inappropriate uses in continuing education of Scriven's (1967) concepts of formative and summative evaluation are described by the author, and guidelines are presented to suggest when formative and summative evaluations are appropriate in continuing education programs. After clarifying what is meant by formative and summative evaluation, the author describes and explains the role of confirmative evaluation, an evaluation analagous to formative and summative evaluation but pertaining to programs...
Topics: ERIC Archive, Adult Education, Adult Programs, Evaluation Methods, Evaluation Needs, Formative...
Valuing in evaluation encompasses two distinct senses of the word, denoted by the terms merit and worth. Merit may be defined as an entity's inherent, intrinsic, context-free value, while an entity's worth is defined as its contextually determined, place-bound value. Determining an entity's merit may take place whenever a number of experts are assembled. Worth can only be determined by viewing the entity in operation or on site. Thus, while merit may be determined in any number of ways, worth...
Topics: ERIC Archive, Conceptual Schemes, Cultural Pluralism, Evaluation, Evaluation Criteria, Evaluation...
The Commission on No Child Left Behind (the Commission), a bipartisan, independent commission formed to develop recommendations for the upcoming reauthorization of the No Child Left Behind Act, released its report on February 13, 2007. The Commission spent the past year traveling across the country, hearing from students, educators, parents, administrators, state and district officials, experts and policymakers in order to develop its report, which provides 75 recommendations. For more...
Topics: ERIC Archive, Federal Legislation, Learning Disabilities, Formative Evaluation, Advisory...
Noting that formative research on educational computer software has traditionally focused on comprehensibility, appeal, and usefulness, this report argues that these three elements tell us very little about the actual role that the formative researcher plays in the instructional design process. This paper outlines the complexity of the tasks and skills required in formative research by presenting a case study of the design of an educational computer program, "Maya Math," one of the...
Topics: ERIC Archive, Case Studies, Computer Assisted Instruction, Courseware, Elementary Education,...
The booklet describes the conceptual approach and operation of the evaluation plan used by Project CAST (Community and School Together), a community-ased career education program for handicapped students. Three levels of evaluation are proposed for providing data for both formative and summative evaluation; these levels are: (1) effort evaluation, (2) performance evaluation, and (3) outcome evaluation. Sample evaluation strategies described include summative evaluation of student outcomes, such...
Topics: ERIC Archive, Career Education, Community Resources, Formative Evaluation, Mild Disabilities,...
This annotated bibliography, one of seven related documents, contains 47 selected references for planning and implementing a comprehensive evaluation of a school program. The references are organized according to problems commonly encountered by school staffs undertaking program evaluations: evaluation of innovative practices, using checklists, formative or process evaluation, general evaluation issues, and methodological issues. (Author)
Topics: ERIC Archive, Administrator Guides, Annotated Bibliographies, Check Lists, Formative Evaluation,...
Reviewed is the development and evaluation of the instructional quality inventory (IQI), a systematic methodology for reviewing the three major products of the Instructional Systems Design process--objectives, test items, and instruction--before conducting student tryouts. The empirically based instructional design support system aids developers in choosing instructional alternatives based on cost/benefits and specific resource limitations. The objective of this report, which is intended for...
Topics: ERIC Archive, Evaluation Methods, Formative Evaluation, Instructional Development, Instructional...
To obtain from students information for formative evaluation of instruction, course-specific questionnaires were developed with individual faculty. Forms were introduced by a statement that the purpose of evaluation was to improve instruction. Although items varied with the course, all had potential action implications and all requested recommendations for change. Results supporting the effectiveness of course-specific evaluation are reported. Information derived instigated responsive faculty...
Topics: ERIC Archive, Formative Evaluation, Higher Education, Measurement Instruments, Questionnaires, Self...
The action evaluation process at a mediation agency in Dayton, Ohio, that offers the services of trained volunteers is described. Action evaluation is an integrated research and intervention methodology for helping to define, promote, and assess success in an endeavor, while using evaluation as a bridge between research and practice in ways that promote greater quality in both. The action evaluation process includes: (1) a baseline stage that incorporates a systematic process for cooperative...
Topics: ERIC Archive, Evaluation Methods, Formative Evaluation, Intervention, Organizational Development,...
