10. Establishing Passing Standards

Authors

Craig N. Mills, Educational Testing Service

Date of this Version

1995

Document Type

Article

Citation

Licensure Testing: Purposes, Procedures, and Practices, ed. James C. Impara (Lincoln, NE: Buros Institute of Mental Measurements, University of Nebraska-Lincoln, 1995).

Comments

Copyright © 1995 by Buros Institute of Mental Measurements. Digital Edition copyright © 2012 Buros Center for Testing.

Abstract

INTRODUCTION

When tests are used to determine eligibility for a license, a passing standard or cut score must be established that divides the test scores into two categories: eligible for license or not. Standard setting has been widely researched and there are many reviews available (see, for example, Jaeger, 1989; Mills & Melican, 1988; Berk, 1986; Hambleton, 1980; Hambleton & Eignor, 1980; and Shepard, 1980a, 1980b), yet there is limited practical advice available for conducting standard setting studies and establishing standards. The one available resource (Livingston & Zieky, 1982) is somewhat dated. The purpose of this chapter is to provide a practical discussion of the entire standard setting process². The steps in a standard setting study are explained. Commonly used standard setting methods are described, examples are provided, and the methods are critiqued. Procedures for conducting a standard setting study and adjusting the resulting preliminary standard are also explained. The chapter also discusses factors other than test performance that can be considered in setting standards on licensure tests.

Standard setting is a multiple-step process involving different groups. There are typically three groups involved in the process: the "test sponsor," the investigator, and expert raters (or judges). The term "test sponsor" refers to the organization (e.g., licensure board), that has ultimate responsibility for the testing program. Although the sponsor may contract for testing services (test development, administration, and statistical analysis), it bears responsibility for the soundness of the test and testing program and has policy, financial, and legal responsibility as well. The investigator is the individual (or group) responsible for conducting the standard setting study and advising the test sponsor on all aspects of it. The investigator may be an employee of the sponsor, a testing services provider, or an independent consultant. The investigator's responsibilities extend from initial discussion of the design of the study through the actual data collection and analysis, and extend (typically) to acting as a resource during the deliberations leading to the establishment of the operational standard. Expert raters are typically educators and/ or practitioners in the field who are convened on one or more occasions to provide judgments about the test, examinees, and (possibly) the appropriateness of the recommended standard.

It is important to identify clearly which parties are involved in each step and what their specific responsibilities are. For example, test sponsors will often use an external investigator to conduct the standard setting study. This is sound practice if standard setting expertise is not available within the sponsor's organization, but does not exempt the sponsor from the responsibility of establishing the final standard. Figure 1 lists the steps in establishing a standard and the parties involved in each step. Each step is explained in the remainder of the chapter.

DETERMINE THE NEED FOR A STANDARD

In most licensure settings, the decision to develop a test is based on the need to make decisions about individuals (e.g., the individual has sufficient knowledge and skills to receive a license or not). However, it is important that the development of the licensure test itself is justified. It is appropriate, for example, for a legislative body to decide that there is sufficient risk to the public from ill-prepared practitioners that a test to distinguish between individuals who can provide appropriate service and those who cannot is necessary.

Livingston and Zieky (1982) suggest that test sponsors be prepared to justify the use of a standard. Although it may be true that fairer licensure decisions will result from the program than from a case-by-case consideration of applications, it is likely that there will be resistance to the imposition of a test. Test sponsors should know the likely criticisms and be ready to respond to them and contrast the fairness of the program with current practice. Several other issues should be considered as well. The sponsor should ensure that the appropriate reliability and validity analyses will be conducted. Administrative procedures should be addressed. For example, how often will individuals be allowed to test? Will periodic license renewal be required? Will current practitioners be "grandfathered" into the program? Under what conditions (if any) should exceptions be granted? How much advance notice will be given of the requirement to pass the test? These issues are more directly related to the operational aspects of the testing program than to the establishment of a standard, but can affect standards. Interested readers are referred to Livingston and Zieky (1982) for a discussion of these and other issues.