5. Systematic Item Writing And Test Construction

Authors

Anthony LaDuca, National Board of Medical ExaminersFollow
Steven M. Downing, American Board of Internal MedicineFollow
Thomas R. Henzel, American Board of Internal Medicine

Date of this Version

1995

Document Type

Article

Citation

Licensure Testing: Purposes, Procedures, and Practices, ed. James C. Impara (Lincoln, NE: Buros Institute of Mental Measurements, University of Nebraska-Lincoln, 1995).

Comments

Copyright © 1995 by Buros Institute of Mental Measurements. Digital Edition copyright © 2012 Buros Center for Testing.

Abstract

Standardized objective testing remains the most popular mode of licensure testing. Even where other types of tests are incorporated, it is often the case that they are provided as complimentary to standardized, multiple-choice (MC) tests. Moreover, scoring theories and standard-setting procedures have been developed over the years in the context of standardized MC testing. At the same time, critics have pointed to limitations of contemporary MC testing practices, including lack of fidelity to real-life challenges and emphasis on recall of factual minutiae. In our view, testing professionals should make conscientious attempts to modify test development procedures so as to address valid criticisms. In this chapter we offer several suggestions for improving licensure test development, although it may not be feasible to adopt the entire array of recommendations we make. We are providing an intentionally wide selection in the hope that testing professionals will find something of use in their field of practice. Our discussion emphasizes careful design and systematic item-writing methods. We describe types of test items and make suggestions for development and maintenance of an item pool. Later we discuss test-construction procedures.

OPERATIONAL ASSUMPTIONS

We assume that the testing program is intended for use in licensing persons who are entering an occupation or profession in a U.S. jurisdiction. Our discussion assumes further that the program is new; however, the implications for already established licensure programs may be clear to the reader. The testing programs we consider are those that rely on paper-and-pencil techniques generally associated with standardized testing. These imply having examinees fill in spaces on answer sheets that are optically scanned at a later time. We are also assuming that the standards for passing the licensure test will be established using one or more of the content-based approaches that are presently available. Such standards are fixed and maintained through equating procedures using the appropriate statistical methods. Details of these procedures are provided elsewhere in this volume. In this chapter we assume that systematic pretesting of newly written multiple-choice questions (MCQs) will be implemented as part of the testing program.

Much of our experience has been in the context of licensing and certifying physicians and our examples are largely restricted to medical applications. We believe that the features we outline will be effective with nonmedical professions as well.

IMPORTANCE OF TEST DESIGN

Test development comprises the full array of activities associated with bringing a standardized assessment into operation. The particulars of what we designate as design are of special significance in development of licensure tests for two reasons. First, the imperative to assemble evidence in support of the content validity of the examination is heightened in the licensure context. Second, the logical and procedural linkages between the design and the test items must withstand close scrutiny.

Job Analysis, Job Relevance and Content Validity

Content validity retains a somewhat controversial character among measurement specialists. Much contemporary commentary relegates content validity to an inferior status because it is described as emerging from the apparent fit between the test content and the persons (i .e., experts) involved in the development of the test. This version of content validity places it outside the preferred paradigm of interpretations of examinee scores. In our view this disparagement of content validity is unwarranted in licensure testing. Validation of licensure tests may rely heavily on evidence of unimpeachable "job relevance" of test content, but there is no reason to exclude empirical processes from content validation, including interpretations of scores. More to the point, the imperative to establish the unimpeachable job relevance of the licensure test enhances the importance of design because it is at the level of test design that the issue of relevance is first addressed.

The job relevance perspective implies that the test items in the licensure examination must be linked through systematic means to a well-defined representation of the demands of the occupation or profession. The Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association & National Council on Measurement in Education, 1985) call for a "job analysis" in licensure test development (Fine, 1986) and this has come to be a well-accepted element of the process (see chapter 4). Although we prefer an alternative method to conventional job analyses, the more significant point is the imperative to start with a representation of the target occupation or profession. The purpose of such a representation is to establish a definition of knowledge and skill that is essential to competent practice. It is possession of the candidate's knowledge and skill that the licensing examination is intended to establish or confirm, and the presumption is that the public is protected by such an assessment.