Copyright © 1984 by Lawrence Erlbaum Associates, Inc. All rights reserved. No part of this book may be reproduced in any form, by photostat, microform, retrieval system, or an y other means without the prior written permission of the publisher.

Lawrence Erlbaum Associates, Inc., Publishers
365 Broadway
Hillsdale, New Jersey 07642

Library of Congress Cataloging in Publication Data
Main entry under title:

Social and technical issues in testing.

(Buros-Nebraska symposium on measurement & testing)
Includes bibliographies and indexes.

A-Z (3 pages)

A

Achievement, 129, 130, 155
Achievement-saturated intelligence tests , 53-54
Alternative choice items, 148- 149 Examples, 150, 151
American Civil Liberties Union, 94
American College Testing Programs (ACT), 18
Applied performance testing , 147
Aptitude, 129, 130, 167
AQ (Achievement or accomplishment quotient) , 5, 130, 131

...

W, Z

Waisman Center, 14
Zenotype, 19

A-Z (6 pages)

Numbers in italics indicate pages with complete bibliographic information.

A
Acker, S. R., 65 , 86
Ackoff, R. L. , 21, 35
Adair, F. L., 122, 125
Adkins, D. c. , 148 , 153
Aisner, D. J ., 32, 35
Alderman, D. L. , 148 , 153
Anastas i, A., 9 1, 99, 107, 129 , 130, 132, 134, 139, 164 , 168 , 169
Anderson, G. E. , Jr. , 2 1, 35
Anderson , 1. R. , 46 , 57
Anderson, R. C. , 142, 153, 157, 169
Arnoff, S. L., 2 1, 35
Ausubel, D. P., 165, 169

...

Z

Zedeck, S., 72, 86
Ziskin , J. , 96, 109

STRUCTURES OF KNOWLEDGE AND ABILITY

Educational achievement refers to what an individual knows and can do in a specified subject area. At issue is not merely the amount of knowledge accumulated but its organization or structure as a functional system for productive thinking , problem solving , and creative invention in the subject area as well as for further learning. The individual’s structure of knowledge is a critical aspect of educational achievement because it facilitates or hinders what he or she can do in the subject area. What a person can do in an area includes a variety of area specific skills, such as extracting a square root or parsing a sentence or balancing a chemical equation , but also broader cognitive abilities that cut across subject areas, such as comprehension, memory retention and retrieval, reasoning, analysis and restructuring, evaluation or judgment, and fluency.

These broader cognitive abilities contribute to the assembly and structuring of knowledge, to the continual reassembly and restructuring of cumulating knowledge, to the accessing and retrieval of knowledge, and to its use in problem representation and solution. "Thus achievement," in Snow's (1980a) words, “is as much an organization function as it is an acquisition function. And new achievement depends as much on transfer of such organization as it does on transfer of specific prior facts and skills [p. 43]. “Because cognitive abilities play a central role in both the acquisition and organization functions of educational achievement, their influence can hardly be suppressed or ignored in educational achievement testing that assesses knowledge structures. However, their role may be reduced in low-level achievement testing that stresses amount of information alone. Let us next consider the nature of developed knowledge structures in more detail and then the nature of developed abilities, before attempting to relate this formulation to other conceptions of educational achievement.

The solution proposed for the second problem is to present the examinee with a series of incomplete propositions, accompanied by two or more alternative completions, only one of which makes the proposition true. Many of the current issues in the writing of achievement test items are related to these two proposed solutions.

A CONCEPTION OF KNOWLEDGE

Knowledge originates in information that can be received directly from observations or indirectly from reports of observations. These observations may be external (objects or events) or internal (thoughts and feelings) (Scheffler, 1965, p. 137). Information feeds the mind and, like food for the body, it must be digested and assimilated. Thinking is the process by which these things can be accomplished (Newman, 1852, p. 134). Information that is simply stored in memory remains only information, the lowest, least useful form of knowledge. But if the information becomes the subject of reflective thought, if those who received it ask themselves, "What does it mean?" "How do we know?" "Why is it so?” the information may come to be understood. It may be integrated into a system of relations among concepts and ideas that constitutes a structure of knowledge. This has been referred to as "semantic encoding" (Anderson, 1972, p. 146). Information that is understood, that is incorporated into a structure of knowledge, tends to be more powerful, more useful, and more satisfying. It is likely to be a more permanent possession than information that is simply remembered (Boulding, 1967, pp. 7- 8).

The basis for verbal knowledge exists in the mind in a form that Polanyi (1958) has called "tacit knowledge." In that form, it is a purely private possession. But if concepts can be abstracted from these images and expressed in words, and if the relations among the concepts can be expressed in sentences, then tacit knowledge is converted into verbal knowledge. This can be communicated and thus made public. It can also be recorded and stored for future reference. It can be manipulated in the processes of reflective thinking. It is thus a very powerful form of knowledge. The peculiar excellence of human beings among all other creatures on earth is their ability to produce and to use verbal knowledge. Thinkers produce it. Teachers and students, planners, and managers use it. Classrooms and libraries and study rooms are full of it. So are conference rooms, memoranda, and reports. It would be difficult to overstate the importance of structures of verbal knowledge in human affairs (Hayakawa, 1941, pp. 15-25; Langer, 1957, pp. 200- 204).

Let us examine specifically the traditional distinction between aptitude and achievement tests. Aptitudes are typically defined more precisely than intelligence, to designate more narrowly limited cognitive domains. Nevertheless, like intelligence, they have traditionally been contrasted with achievement in testing terminology. This contrast dates from the early days of testing, when it was widely assumed that achievement tests measured the effects of learning, whereas intelligence and aptitude tests measured so-called innate capacity, or potentiality, independently of learning. This approach to testing in turn reflected a simplistic conception of the operation of heredity and environment that prevailed in the j 920s and 1930s. The relevant historical background has been thoroughly examined in a recent book by a science historian, Hamilton Cravens, which covers the heredity- environment controversy among American scientists between the two World Wars (Cravens, 1978; see also Anastasi, 1979).

HISTORICAL ANTECEDENTS

Common misconceptions about the relation between aptitude and achievement tests are highlighted by an index introduced in the 1920s and variously named an achievement quotient or an accomplishment quotient. Both terms having the same initials, this index soon came to be known as the AQ. Its origin is generally attributed to Raymond Franzen (1920, 1922). The AQ could be found by dividing the individual’s educational quotient (EQ) by his or her intelligence quotient (IQ). The EQ was the ratio of educational age (EA) to chronological age (CA). The AQ could also be computed more directly by dividing educational age by mental age. The educational age was found by referring the score on an achievement battery to the age norms for that battery. Still another procedure was to use age norms for tests in particular academic subjects, like reading or arithmetic, to find " subject ages" for the individual, and then to average these subject ages to obtain the educational age.

Early textbooks on testing regularly included a discussion of the AQ as a means of evaluating a student's educational performance in relation to that student's intellectual potential- a means of comparing achievement with capacity to learn (Freeman, 1926, 1939; Garrett & Schneck, 1933; Greene, 1941; Lincoln & Workman, 1935; Mursell, 1947). It is interesting to trace the statements about the AQ in texts appearing from the 1920s to the 1940s and early 1950s. Even the earliest discussions called attention to the technical and statistical weaknesses of the AQ as a ratio. The major criticisms fell into two categories: The first category was similar to the now familiar criticisms of the traditional ratio IQ; the second was similar to the equally familiar criticism of grade norms--educational age norms were certainly no better than educational grade norms.

Within such a context as that just described, the Buros Institute of Mental Measurements has always played a unique role. The science of measurement or measurement theory has not been one of the Institute's chief concerns, although the Institute is often an indirect beneficiary of such contributions. However, the Institute has had major involvement with the evaluation of test products, the products of an applied science, and with the education of test users in the more effective selection and use of those products. Because of the nature of this involvement, the Institute has had a perspective on the three separate aspects of the field of measurement that is not typical of those representing only the singular aspects of the continuum. It is this unique perspective of the Institute that will serve as the distinguishing feature of the discussion to follow.

The purpose of this discussion is to evaluate critically the contributions and progress made in these separate, but interrelated, aspects of measurement: knowledge, implementation, and use. The theme of this discussion is that the greatest progress has been made in our knowledge, lesser progress in implementation, and the very least progress in selection and effective use. The implication of the discussion is that there is a pressing need to redress the imbalance that has developed.

Tests themselves, by and large, are facially neutral. They do not inherently discriminate against those who take them and, undoubtedly, scores derived from tests have been used to admit, advance, and employ. For most people, however, test results have served as exclusionary mechanisms- to segregate, institutionalize, track, and deny access to coveted and increasingly scarce employment opportunities.

At one time, the work of academic and applied psychometricians went virtually unexamined by the law, but as the use of tests increased in the United States, so did their potential for causing legally cognizable injury to test takers. As a result, there is probably no current activity performed by psychologists so closely scrutinized and regulated by the legal system as testing.

In today's environment, whether the validation is intended for employee selection, educational decisions, or personal counseling, there is an increasing probability that the outcomes of research will have legal implications. In the past, a testing program could be set up in terms of professional judgment without including the experimental validation of the procedures. If the individuals involved in establishing the test program were knowledgeable, it was quite possible the tests, although unvalidated, would make a practical contribution in terms of the goals intended. In the absence of a formal validation, however, one would never know the extent to which the testing program was successful or superior to another assessment procedure. A testing program that does not involve validation research is at best an unknown and at worst may be an outright fraud. In either case, the likelihood that testing procedures will have to be defended, including the possibility of legal action, has increased dramatically.

The purpose of the present review is to look carefully at the current status and future directions of test validation research. It will be of value to look at what we know, some of the problems with the process by which tests have been validated up to now, what needs to be learned, and how we will move ahead in the area of test validation research. Finally, it will be important to consider test validation research as a vehicle for improving test construction and test usage.

I have divided my analysis of the contribution of cognitive psychology to test construction into four main parts dealing, respectively, with the contributions of cognitive psychology to: (1) content for construction of tests; (2) validation of tests; (3) scoring and interpretation of tests; and (4) modification of tests. Before discussing these contributions, however, let me say just what are the characteristics that define "cognitive psychology" and what psychologists do in the cognitive- psychological investigation of intelligence.

Cognitive psychology is the study of the mind in terms of the mental (cognitive) representations and processes that underlie observable behavior. In particular, I find that cognitive researchers tend to address five main questions:
1. What are the mental processes that constitute intelligent task performance?
2. How rapidly and accurately are these processes performed?
3. Into what strategies for task performance do these mental processes combine?
4. Upon what forms of mental representation do these processes and strategies act?
5. What is the knowledge base that is organized into these forms of representation, and how does it affect and become affected by the processes, strategies, and representations that individuals use?

These questions have been asked of performance on a rather wide range of cognitive tasks.

How new all this field really is: According to Stanley and Hopkins (1972, p. 163), the first large-scale testing was done in the City of New York Survey, in 1911. Oscar Buros was 6 years old then, so we can think of most of the astonishing developments in measurement really happening during his lifetime. And the first machine for scoring of answer sheets, the old IBM 805, was developed when Oscar was 30. Many of us can remember, only 20 years ago, many clerical workers reading the dials from these machines and writing the scores as they might be estimated from this analog device. Then these tools also became obsolete as the field was overtaken by optical readers and computer scoring. So Oscar and Luella Buros have witnessed the explosion of testing into a central institution of education, of psychology, of all the social and behavioral sciences. But they have done much more than witness: Their publications have served as a steady center of this growth, and their independence has established a tradition of reputation and honor as a goal, if not always as a realization, of the profession and the practice of testing.

The establishment of such published symposia from the Buros Institute is an important further step. There is a major place for such a forum. I hope these symposia will represent a determined effort to stand apart from the testing giants, just as Buros did, and to remain independent of federal agencies as well. The Institute, and these symposia, should continue to sponsor solid, sometimes severe criticism of tests and test practices, also as Buros did. They should similarly stand apart from the political huckstering and trend riding, the cheap shots against testing, and apart from the constant distortion of what tests tell us about ourselves and our world.

Of course, the Institute should make full modern use of word-processing, automatic mailing, information retrieval, and all the present and future efficiencies of operation becoming available. But hopefully there will remain these steady principles that marked Buros' work, and a similar vision of mental measurement, of how it can help our society to be happier and more productive.

Some of the questions being asked by the public are value laden, providing topics for many thoughtful but heated debates. For example: Would we be better off as a society if we did not have tests? Should testing be banned? Other questions are technical in nature and require accurate answers from the measurement community, which communicates to the public the present state of the art in measurement, assessment, and interpretation. Finally, questions such as "How can tests be used to eliminate the errors made in the selection process?" can provide an impetus within the measurement field for both theoretical and empirical development and yet are not ones that can, at least so far, be definitely answered .

The measurement field should take serious stock of itself and assess, as well as possible, the boundaries of its capabilities. From this assessment, it would be possible to communicate with the public about what testing can do, may be able to do, and is incapable of ever doing. At the present time, however, there appears to be an informational and expectational gap concerning what can be possible with the use of test results. Unless measurement experts and test users obtain a direct line to the angels, for example, error-free measurement will never be a reality!

Part of the communication and expectation gap can be assigned to a lack of measurement sophistication on the part of the public. Measurement course work and classes are not readily accessible to the public as a whole and may not be truly meaningful and usable to the public even if they were . A well-meaning but confused public provides fertile ground for test misunderstanding and misrepresentation by both knowledgeable and unknowledgeable test representatives. Tests enter into the lives of the public in so many ways; questionnaires, market surveys, school achievement batteries, classroom exams, and admission screenings are only a few possibilities. Yet the knowledge level of the public is minimal at best with regard to test information and interpretation.

The interplay between testing and society has resulted in both praise and criticism from concerned citizens, psychologists, educators, and numerous other professional and consumer groups. For over 40 years, Oscar K. Buros, as Director of The Institute of Mental Measurements and Editor of the Mental Measurements Yearbooks. contributed immensely to this interplay between testing practices and societal issues. On March 19, 1978, Oscar Buros died. Luella Buros , his wife and lifelong helpmate, completed the work on The Eighth Mental Measurements Yearbook with the support of the Institute's devoted staff. She also took steps to relocate the Institute to ensure the continuation of the Institute's scholarly work and services for test consumers. The new Buros Institute of Mental Measurements is now at the University of Nebraska- Lincoln and is under grant from The University of Nebraska Foundation.

An important objective of the new Buros Institute is to conduct an extended outreach effort that will help communicate more effectively with test users about contemporary issues in testing . Thus, it was the combination of recent social issues focusing on testing and our desire to fulfill more vigorously the mission of the Buros Institute that motivated the development of an annual scholarly symposium and this series on measurement and testing.

We intend each symposium and volume in this series to present state-of-the art knowledge that will contribute to the improvement of test construction and test usage. Such a schema will incorporate topics across a broad spectrum such as theoretical models of human behavior, test standardization procedures, social and legal factors in testing, administration of testing programs , and test-based decision making. Thus, the series will be focused thematically and yet be flexible enough to integrate current and future measurement and testing issues into its schema.

The success of our first Buros- Nebraska symposium and this volume is the result of the efforts of many individuals. We thank Luella Buros for having faith in us to carryon and extend a tradition that has become so important to the measurement field and to test users. Barbara Plake , as editor of the first volume in the series, made conceptual and editorial contributions that were of critical importance to its success. Finally, we want to thank Larry Erlbaum for his support, encouragement, and commitment to the project and to its timely completion.

Preface

1. Filling the Gaps Between Test Outcomes and Usage: An Introduction Barbara S. Plake Purpose of the Volume Overview of the Chapters Conclusions References

PART I SOCIAL AND TECHNICAL INFLUENCES

2. Struggles and Possibilities: The Use of Tests in Decision Making Ellis Batten Page A Double Standard The Value of Testing Ideological and Scientific Issues Decision Making Test Scores and Deeper Values Production Functions and Causal Research Conclusions References

3. What Cognitive Psychology Can (and Cannot) Do for Test Development Robert J. Sternberg What is Cognitive Psychology? Cognitive Psychology and Test Construction Cognitive Psychology and Test Validation Cognitive Psychology, Test Scoring, and Test Interpretation Cognitive Psychology and Test Modification References

4. The Status of Test Validation Research Lyle F. Schoenfeldt Traditional Approaches to Test Validation Construct Validation Multivariate Validation Models Validity Generalization Productivity Analysis Implications for Test Construction and Test Usage References

5. Social and Legal Influences on Test Development and Usage Donald N. Bersoff Social Influences Social Influences on Legal Decisions: Some Examples Social Science Evidence and the Courts Psychologists and Public Policy References

6. Testing and the Oscar Buros Lament: From Knowledge to Implementation to Use James V. Mitchell, Jr. Measurement Theory and Knowledge Test Technology and the Challenge of Implementation The Bottom Line: The Selection and Effective Use of Tests Conclusions and Recommendations References

PART II INFLUENCES ON APTITUDE AND ACHIEVEMENT TEST DEVELOPMENT AND USAGE

7. Aptitude and Achievement Tests: The Curious Case of the Indestructible Strawperson Anne Anastasi Historical Antecedents Debunking Ventures The Continuum of Developed Abilities More about Aptitudes References

8. Achievement Test Items: Current Issues Robert L. Ebel A Conception of Knowledge The Relation of Knowledge to Ability The Measurability of Human Characteristics The Relative Merits of Essay and Objective Tests The Merits of Items Based on Realistic Problem Situations The Merits of Alternate-Choice Items Prospects for a Technology of Item Writing Concluding Statement References

9. Abilities and Knowledge in Educational Achievement Testing: The Assessment of Dynamic Cognitive Structures Samuel Messick Structures of Knowledge and Ability Expertise and Aptitude The Failings of Fallacies References

Author Index

Subject Index