Two mathematical programming approaches are presented for the assembly of ability test from item pools calibrated under a multidimensional item response theory model. Item selection is based on Fisher's Information matrix. Several criteria can be used to optimize this matrix. In this paper, the A-criterion and the D-criterion are applied. In a mathematical programming approach, both criteria provide good results for the two dimensional case. Empirical examples for a two-dimensional mathematics...
Topics: ERIC Archive, Ability, Item Banks, Test Construction, Test Items, Veldkamp, Bernard P.
This guide is divided into four parts. Part I, an introduction, contains a discussion of why item banking has failed thus far to become a widely accepted procedure for test development, as well as some miscellaneous information on item banking that may be useful. In Part 2, completed questionnaires from 22 organizations with item banks give information about content and size of the banks, services available from the sponsoring organization, etc. The results of an item bank telephone survey with...
Topics: ERIC Archive, Educational Testing, Information Sources, Item Banks, Test Construction, Testing...
A project was carried out to determine the degree of content homogeneity that a test item pool must have in order to accomplish successful Rasch calibration. Mathematics item pools were administered to upper elementary children. The items were analyzed under two conditions, with items organized into separate subtests and as a global mathematics test. Results indicate that satisfactory item calibration can be carried out with global mathematics tests, avoiding the necessity of organizing items...
Topics: ERIC Archive, Elementary Education, Item Analysis, Item Banks, Mathematics, Test Construction, Tests
This paper discusses optimal test construction, which deals with the selection of items from a pool to construct a test that performs optimally with respect to the objective of the test and simultaneously meets all test specifications. Optimal test construction problems can be formulated as mathematical decision models. Algorithms and heuristics have been developed to solve the models that can be used to construct tests. (Contains 2 tables, 2 figures, and 27 references.) (Author/SLD)
Topics: ERIC Archive, Algorithms, Item Banks, Selection, Test Construction, Test Items, Veldkamp, Bernard P.
This study explored the dangers of item theft in terms of impact on test quality and fairness. Simulations were used to explore the impact of item theft on test taker scores. In the simulation, a simulated, organized group of thieves took the test, memorized the items received, and distributed the items to future test takers. Impact was explored for varying numbers of thieves and for thieves of two ability levels. Results show that test takers scores can be largely inflated when test takers...
Topics: ERIC Archive, Cheating, Item Banks, Simulation, Test Items, Schnipke, Deborah L., Scrams, David J.
In the context of paper and pencil testing, the frequency of the exposure of items is usually controlled through policies that regulate both the reuse of test forms and the frequency with which a candidate may retake the test. In the context of computerized adaptive testing, where item pools are large and expensive to produce and testing can be on a continual basis, new strategies are required. This paper discusses the popular randomization strategy for controlling item security and a less well...
Topics: ERIC Archive, Adaptive Testing, Computer Assisted Testing, Item Banks, Models, Test Format, Test...
A specific application of the process of automating exams for any introductory statistics course is described. The process of automating exams was accomplished by using the Statistical Test Item Collection System (STICS). This system was first used to select a set of questions based on course requirements established in advance; afterward, STICS was utilized to generate multiple exams from this collection of questions. The procedure for automating exams involved five steps: (1) establishing...
Topics: ERIC Archive, Computer Assisted Testing, Computer Programs, Higher Education, Item Banks,...
Recently, behavioral scientists have examined interpersonal manipulation and its effects on individuals, groups, and society. Richard Christie's description of the Machiavellian personality orientation highlights the behaviors of those individuals who are highly successful manipulators of others. Scanning some Machiavellian works for items to tap the construct, a pool of 71 was constructed by Christie and an attempt is made here to investigate the factorial structure of Machiavellianism using...
Topics: ERIC Archive, College Students, Factor Analysis, Factor Structure, Human Relations, Item Banks,...
Because different types of computerized tests exist and continue to emerge, the term "computer-based testing" does not encompass all of the various models that may exist. As a result, test delivery model (TDM) is used to describe the variety of methods that exist in delivering tests to examinees. The criterion that is used to distinguish between the various TDMs is the extent to which the test is adaptive to an examinee's performance during the test session. On one side of the...
Topics: ERIC Archive, Computer Assisted Testing, Adaptive Testing, Models, Delivery Systems, Item Banks,...
A method is described for simultaneous test construction using the Operations Research technique zero-one programming. The model for zero-one programming consists of two parts. The first contains the objective function that describes the aspect to be optimized. The second part contains the constraints under which the objective function should be optimized. The selection of items is based on information from item response theory. Simultaneous test design is used when tests have to be constructed...
Topics: ERIC Archive, Item Banks, Latent Trait Theory, Mathematical Models, Operations Research, Test...
Projects to develop an automated item banking and test development system have been undertaken on several occasions at the Air Force Human Resources Laboratory (AFHRL) throughout the past 10 years. Such a system permits the construction of tests in far less time and with a higher degree of accuracy than earlier test construction procedures. This paper details Classical Test Theory and Item Response Theory (IRT) approaches to item banking and test construction and their relevance to the...
Topics: ERIC Archive, Automation, Computer Assisted Testing, Item Banks, Item Response Theory, Test...
This paper discusses item banks calibrated to indicate levels of difficulty to assist in test development. The item bank topics discussed are: (1) purpose; (2) development issues; (3) advantages and disadvantages; and (4) practical issues. The most common issues are content validity, reliability, concerns with software purchase and programming, classification and analysis of items, maintenance and upkeep, and good item writing practice. It is important to remember that item response theory...
Topics: ERIC Archive, Item Banks, Item Response Theory, Test Construction, Test Items, Validity, Anzaldua,...
In planning the development of the system for computer assisted assembly of tests, it was agreed at the outset that one of the basic requirements for the successful initiation of any such system would be the development of a detailed item content classification system. The design of the system for classifying item content is a key element in systems development. Without an adequate classification system, it would be impossible to write test specifications and assemble tests that sample required...
Topics: ERIC Archive, Classification, Codification, Computer Oriented Programs, Computer Programs, Item...
A previously developed method of automatically selecting items for inclusion in a test subject to constraints on item content and statistical properties is applied to real data. Two tests are first assembled by experts in test construction who normally assemble such tests on a routine basis. Using the same pool of items and constraints articulated by test construction experts the same two tests are reassembled automatically. The manual and automatic assemblies are compared by test specialists...
Topics: ERIC Archive, Algorithms, Automation, Coding, Computer Assisted Testing, Item Banks, Selection,...
This bibliography includes papers concerned with using a computer to help construct a list of questions for use as a test or exercise. It is not intended to contain references to papers which deal exclusively with computers administering tests. Sixty-four references are provided. (Author/WH)
Topics: ERIC Archive, Bibliographies, Computer Oriented Programs, Computer Programs, Computers, Item Banks,...
The distribution of a certain item response theory (IRT) based person fit index to identify systematic types of aberrance is discussed. For the Rasch model, it is proved that: (1) the joint distribution of subtest-residuals (the components of the index) is asymptotically multivariate normal; and (2) the distribution of the index is asymptotically chi-square. The parameters of these asymptotic distributions depend on whether ability of a person is known or estimated. Furthermore, the rate of...
Topics: ERIC Archive, Ability, Computer Simulation, Foreign Countries, Item Banks, Latent Trait Theory,...
Six methods for assembling tests from a pool with an item-set structure are presented. All methods are computational and based on the technique of mixed integer programming. The methods are evaluated using such criteria as the feasibility of their linear programming problems and their expected solution times. The methods are illustrated for two item pools with a set structure from the Law School Admission Test (LSAT). The methods are: (1) simultaneous selection of items and sets; (2)...
Topics: ERIC Archive, Higher Education, Item Banks, Selection, Test Construction, Test Items, van der...
This brief overview notes that an adaptive test differs from standardized achievement tests in that it does not consist of a certain set of items that are administered to a group of examinees. Instead, the test is individualized for each examinee. The items administered to the examinee are selected from a large pool of items on the basis of the correctness of the examinee's previous responses. Several steps are involved in constructing a computerized adaptive test (CAT): (1) the item pool must...
Topics: ERIC Archive, Adaptive Testing, Computer Assisted Testing, Item Banks, Latent Trait Theory, Test...
An adaptive test can usually match or exceed the measurement precision of conventional tests several times its length. This increased efficiency is not without costs, however, as the models underlying adaptive testing make strong assumptions about examinees and items. Most troublesome is the assumption that item pools are unidimensional. Truly unidimensional item pools are the exception rather than the rule, so procedures have been established for handling multidimensional pools. One option is...
Topics: ERIC Archive, Ability, Adaptive Testing, Computer Assisted Testing, Equations (Mathematics), Item...
A unique system is described for creating tests by computer. It is unique because, instead of storing items in the computer, item algorithms similar to Hively's notion of item forms are banked. Every item, and thus every test, represents a sample from domains consisting of thousands of items. The paper contains a discussion of the special practical applications of such tests, a description of the easy-to-learn user language in which item algorithms are written, and the results of using the...
Topics: ERIC Archive, Computer Assisted Testing, Computer Programs, Criterion Referenced Tests, Item Banks,...
Computer assisted test construction (CATC) is a new testing technique that seems to provide ease and flexibility for faculty members and students. The purpose of this paper was to verify that student test scores are not adversely affected by implementation of CATC. Two sections of a basic course in cataloging were tested for one semester. One section of 30 students was used as a control group while the other 20 students served as the experimental group. The students were pre-tested and then...
Topics: ERIC Archive, Computer Assisted Instruction, Item Banks, Library Education, Pretesting, Pretests...
Many applications of educational testing have a missing data aspect (MDA). This MDA is perhaps most pronounced in item banking, where each examinee responds to a different subtest of items from a large item pool and where both person and item parameter estimates are needed. The Rasch model is emphasized, and its non-parametric counterpart (the Mokken scale) is considered. The possibility of tailoring test items in combination with their estimation is discussed; however, most methods for the...
Topics: ERIC Archive, Adaptive Testing, Educational Testing, Estimation (Mathematics), Foreign Countries,...
The Office of Desegregation of the Little Rock, Arkansas, Public School District was given the responsibility for developing an instrument to evaluate the implementation of educational equity in the district pursuant to federal court desegregation mandates. Office of Desegregation personnel invited in Center for Academic Excellence staff of the University of Central Arkansas to collaborate in constructing the instrument. After outlining the goals of the endeavor, the team agreed to the...
Topics: ERIC Archive, Desegregation Plans, Educational Assessment, Equal Education, Item Banks, Program...
With the advent of computer-based testing (CBT) and the need to increase the number of items available in computer adaptive test pools, the idea of item variants was conceived. An item variant can be defined as an item with content based on an existing item to a greater or lesser degree. Item variants were first proposed as a way to enhance test security by increasing the size of CBT item pools. Variants are now also seen as useful rapid item-creation pools in programs that use paper-based...
Topics: ERIC Archive, Adaptive Testing, Computer Assisted Testing, Item Banks, Test Construction, Test...
This paper presents a new heuristic approach to interactive test assembly that is called the successive item replacement algorithm. This approach builds on the work of W. J. van der Linden (1987) and W. J. van der Linden and E. Boekkooi-Timminga (1989) in which methods of mathematical optimization are combined with item response theory to construct tests from larger collections or pools of items. This new approach is contrasted with two more formal models and other heuristic approaches that...
Topics: ERIC Archive, Algorithms, Automation, Computer Selection, Heuristics, Interaction, Item Banks, Item...
The construction of parallel tests from item response theory (IRT) based item banks is discussed. Tests are considered parallel whenever their information functions are identical. After the methods for constructing parallel tests are considered, the computational complexity of 0-1 linear programming and the heuristic procedure applied are discussed. Two methods for selecting parallel tests in succession (sequential test construction) are formulated. The first uses a non-partitioned item bank...
Topics: ERIC Archive, Heuristics, Item Banks, Latent Trait Theory, Mathematical Models, Simulation, Test...
This document consists of a teacher's guide for a competency-based course on homestead and gardening skills designed for North Carolina's 11th- and 12th-grade students, and a list of competency test items applicable to the course. The teacher's guide contains course specifications, a list of competency statements, a sheet describing each unit of instruction, and a 161-item bibliography. The units cover general concepts in agriculture and agribusiness; Future Farmers of America activities and a...
Topics: ERIC Archive, Agribusiness, Agricultural Education, Agricultural Engineering, Agricultural Skills,...
The purpose of this study was to evaluate both the written Department of Motor Vehicles (DMV) driver licensing test, and a large sample of driver knowledge test items selected from the University of Michigan's Highway Safety Research Institute (HSRI) item pool. Test forms were administered to 48,000 California drivers license applicants. The variables analyzed were subject's sex, age, education, annual mileage, and prior six year driving record in relationship to test form and item scores. New...
Topics: ERIC Archive, Certification, Evaluation, Item Banks, Predictor Variables, Scores, Statistical...
This paper compares the use of multiple pools versus a single pool with respect to test security against large-scale item sharing among some examinees in a computer-based test, under the assumption that a randomized item selection method is used. It characterizes the conditions under which employing multiple pools is better than using a single whole pool in terms of minimizing the expected number of compromised items encountered by an examinee. The results obtained in this paper provide...
Topics: ERIC Archive, Comparative Analysis, Test Items, Item Banks, Computer Assisted Testing, Probability,...
A new test construction method based on integer linear programming is described. This method selects optimal tests in small amounts of computer time. The new method, called the Cluster-Based Method, assumes that the items in the bank have been grouped according to their item information curves so that items within a group, or cluster, are interchangeable. Introducing this assumption may reduce the number of decision variables in the model. Because of the assumption of interchangeability, the...
Topics: ERIC Archive, Computer Assisted Testing, Item Banks, Latent Trait Theory, Mathematical Models, Test...
Designing a test using three-parameter item response theory (IRT) is discussed. A brief review of IRT is followed by a discussion of two types of test design: (1) selecting items using confidence envelopes (confidence envelope method); and (2) using item characteristic curves and their confidence intervals (test envelope method). The confidence envelope method and the test envelope method are evaluated based on their reliability coefficients, using a set of seven items. Results illustrate that...
Topics: ERIC Archive, Ability, Equations (Mathematics), Item Banks, Item Response Theory, Mathematical...
The SOCRATES Computer Assisted Test Retrieval System is a bank of test questions and answers for 11 subject matter areas. The system has been available since 1974 to faculty and students of the California State University and College System. Both batch-mode and interactive versions are available. Telephone requests for test forms or reproduction masters can be made one day and received by courier the following day. Users with data-links to the central processor may have tests printed locally on...
Topics: ERIC Archive, College Instruction, Computer Assisted Instruction, Computer Programs, Higher...
The development of the Personality Inventory for Children (PIC) began with the core concept that maternal reports would provide data for child guidance evaluation and the consequent belief that maternal responses to a 600-item administration booklet could yield scales useful in determining child and family status. Two areas of weakness were found: (1) the lack of depth of clinical correlates left the test user to extrapolate from assumed characteristics of criterion group members and/or scale...
Topics: ERIC Archive, Behavior Problems, Evaluation Methods, Guidance, Guides, Identification, Item Banks,...
This publication encompasses questions for Ceramics, Graphic Arts, Metals, and Plastics for the second of a series. The use of this publication and the previously published (1973) book containing resource items for Drawing, Electricity/Electronics, Power Mechanics, and Woods (ED 109 457) will provide complete coverage of the basic series courses as outlined in "A Handbook for Administrators" (1970). These two publications are designed to assist in the improvement of classroom tests at...
Topics: ERIC Archive, Ceramics, Educational Media, Graphic Arts, Industrial Arts, Item Banks, Metals,...
Several methods have been developed for use on constrained adaptive testing. Item pool partitioning, multistage testing, and testlet-based adaptive testing are methods that perform well for specific cases of adaptive testing. The weighted deviation model and the Shadow Test approach can be more generally applied. These methods are based on different ideas about how to optimize the performance of computerized adaptive testing (CAT), and they have different advantages. In this paper, both ideas...
Topics: ERIC Archive, Adaptive Testing, Computer Assisted Testing, Item Banks, Test Construction, Test...
The central idea in building and maintaining an item bank is to calibrate all the items onto a "common variable." The arithmetic involved in the calibration process is presented. It is recommended that an analysis of fit be done in every application to verify that the estimates of item difficulties are in fact sample-free. These procedures are explained. Once an item bank is built, a common calibration for all items should be established and routinely checked. Special procedures for...
Topics: ERIC Archive, Equated Scores, Goodness of Fit, Item Banks, Latent Trait Theory, Test Construction,...
The current status of banks of test items existing across the United States was determined through a survey conducted between September and December 1987. Item "bank" in this context does not imply that the test items are available in computerized form, but simply that "deposited" test items can be withdrawn for use. Emphasis was on documenting the sources of test items in the various subject areas that respondents considered to measure higher order thinking skills. Complete...
Topics: ERIC Archive, Adaptive Testing, Computer Assisted Testing, Item Banks, National Surveys, Test...
It is proposed that the assembly of tests for the measurement of multiple abilities be based on targets for the (asymptotic) variance functions of the estimators in each of the abilities. A linear programming model is presented that can be used to computerize the assembly process. Several cases of test assembly dealing with multidimensional abilities are distinguished, and versions of the model applicable to each of these cases are discussed. An empirical example of a test assembly program from...
Topics: ERIC Archive, Ability, Estimation (Mathematics), Foreign Countries, Item Banks, Measurement...
Recent advances in psychometrics and computer technology encourage the development of model-based methods of individualized testing on a microcomputer, where each examinee receives short tests and where the number of pretest items that can be administered is severely restricted. On-line (i.e., data is collected on operational equipment) methods for calibrating pretest items in this setting face new challenges. In the context of adaptive testing (CAT), two LOGIST-based methods of on-line...
Topics: ERIC Archive, Adaptive Testing, Computer Assisted Testing, Item Banks, Item Response Theory, Online...
Modified Parallel Analysis (MPA) is a heuristic method for assessing "approximate unidimensionality" of item pools. It compares the second eigenvalue of the observed correlation matrix with the corresponding eigenvalue extracted from a "parallel" matrix generated by a unidimensional and locally independent model. Revised Modified Parallel Analysis (RMPA) generalizes MPA and alleviates some of its technical limitations. An important and useful feature is a new method for...
Topics: ERIC Archive, Equations (Mathematics), Heuristics, Item Analysis, Item Banks, Matrices, Models,...
The Objective-Item Bank presented covers 16 sections of four subject areas in each of four grade levels. The four areas are: Language Arts, Math, Social Studies, and Science. The four grade levels are: Primary, Intermediate, Junior High, and High School. The Objective-Item Bank provides school administrators with an initial starting point for curriculum development and with the instrumentation for program evaluation, and offers a mechanism to assist teachers in stating more specifically the...
Topics: ERIC Archive, Behavioral Objectives, Curriculum Development, High Schools, Individualized...
A common practice in some certification fields (e.g., information technology) is to draw items from an item pool randomly and apply a common passing score, regardless of the items administered. Because these tests are commonly used, it is important to determine how accurate the pass/fail decisions are for such tests and whether fairly small, simple changes can be made to such tests to improve their psychometric properties without being too large a burden on the testing program. This simulation...
Topics: ERIC Archive, Decision Making, Difficulty Level, Item Banks, Pass Fail Grading, Simulation, Test...
The Objective-Item Bank presented covers 16 sections of four subject areas in each of four grade levels. The four areas are: Language Arts, Math, Social Studies, and Science. The four grade levels are: Primary, Intermediate, Junior High, and High School. The Objective-Item Bank provides school administrators with an initial starting point for curriculum development and with the instrumentation for program evaluation, and offers a mechanism to assist teachers in stating more specifically the...
Topics: ERIC Archive, Behavioral Objectives, Curriculum Development, Individualized Instruction, Item...
This digest presents a list of recommendations for writing multiple-choice test items, based on psychometrics statistics are typically provided by a measurement, or test scoring, service, where tests are machine-scored or by testing software packages. Test makers can capitalize on the fact that "bad" items can be differentiated from "good" items psychometrically. Tests can be improved by maintaining a pool of "good" items from which future tests can be drawn. Once...
Topics: ERIC Archive, Item Analysis, Item Banks, Measurement Techniques, Multiple Choice Tests,...
In order to verify that a test made of items selected from a pool produces the same Rasch scaled achievement scores as would be predicted from previous Rasch difficulty calibrations for those items, the following experiment was conducted. Seven math tests with similar content but graduated difficulty were administered to students in grade seven. The tests overlapped each other so that each test included about 20 items that were also included in one of the other six tests. The items in these...
Topics: ERIC Archive, ERIC, Doherty, Victor W. Forster, Fred Correlation, Item Banks, Prediction, Raw...
The Objective-Item Bank presented covers 16 sections of four subject areas in each of four grade levels. The four areas are: Language Arts, Math, Social Studies, and Science. The four grade levels are: Primary, Intermediate, Junior High, and High School. The Objective-Item Bank provides school administrators with an initial starting point for curriculum development and with the instrumentation for program evaluation, and offers a mechanism to assist teachers in stating more specifically the...
Topics: ERIC Archive, Behavioral Objectives, Curriculum Development, Elementary Education, Individualized...
For almost 20 years, the Portland public schools have maintained a testing program anchored in locally constructed tests. The district's newly developed tests are called goal-referenced tests, a version of criterion-referenced tests, that meet the district's purposes because they reflect what well-qualified teachers in the district believe should be taught and measured. In constructing the tests, curriculum personnel select goals they believe the test should measure; teachers develop items that...
Topics: ERIC Archive, Criterion Referenced Tests, Educational Objectives, Elementary Secondary Education,...
Test security has been a concern for computerized adaptive tests (CAT) due to the nature of continuous testing. This concern becomes unprecedentedly severe with increasingly easy access to the World-Wide-Web where some examinees post on the internet their recollections of items they are administered, leaving future examinees with opportunities to get their scores inflated by having pre-knowledge of the items that appear in their tests. This study investigated, given some items in the item pool...
Topics: ERIC Archive, Adaptive Testing, Computer Assisted Testing, Test Items, Item Banks, Disclosure,...
The Objective-Item Bank presented covers 16 sections of four subject areas in each of four grade levels. The four areas are: Language Arts, Math, Social Studies, and Science. The four grade levels are: Primary, Intermediate, Junior High, and High School. The Objective-Item Bank provides school administrators with an initial starting point for curriculum development and with the instrumentation for program evaluation, and offers a mechanism to assist teachers in stating more specifically the...
Topics: ERIC Archive, Behavioral Objectives, Curriculum Development, Elementary Education, Individualized...
Computerized adaptive testing (CAT), which was first developed four decades ago, begins with a large pool of questions and then selects individual questions for test takers, depending on their responses as they progress. As the test taker answers questions correctly, the questions become more difficult. As the test taker answers incorrectly, the questions become less difficult. CAT requires the following components: a pool of questions to draw from, calibrated to a common measurement scale; a...
Topics: ERIC Archive, ERIC, Computer Assisted Testing, Adaptive Testing, Item Banks, Barriers, Affordances,...
