Probing Question Order Effect in Chemistry Concept Inventories

Authors

Molly Undersander, University of Nebraska - LincolnFollow
Travis J. Lund, Oregon Institute of TechnologyFollow
Laurie S. Langdon, University of Colorado BoulderFollow
Marilyne Stains, University of Nebraska - LincolnFollow

Date of this Version

4-2016

Document Type

Poster

Citation

Undersander, M., Lund, T., Langdon, L., and Stains, M., 2015, Probing Question Order Effect While Developing a Chemistry Concept Inventory: Manuscript submitted for publication.

Comments

Copyright (c) 2016 Molly Undersander, Travis J. Lund, Laurie Langdon, and Marilyne Stains

Abstract

The design of assessment tools is critical to evaluate accurately students’ understanding of chemistry. Although extensive research has been conducted on various aspects of assessment tool design, few studies in chemistry have focused on the impact of the order in which questions are presented to students on the measurement of students’ understanding and students’ performance. This potential impact has been labeled the question order effect in other literature. The set of studies described in this article tested whether question order effects were present within a concept inventory on acid-based chemistry. In particular, we tested whether the order of two conceptually isomorphic questions, one pictorial and one verbal, affected students’ performance on the concept inventory. Two different versions of the inventory were developed and collected from students enrolled in the second semester of first-year university chemistry courses (general chemistry; N=774) at two different institutions and to students enrolled in the first semester of organic chemistry (N=163) at one of the two institutions. Students were further divided in two groups based on their self-reported level of effort in answering the concept inventory. Interviews were also conducted with a total of 26 students at various stages of the studies. Analyses of differences in students’ responses to the two versions of the inventory revealed a question order effect on the pictorial question at one institution and on the verbal question at the other institution. These mixed findings agree with prior studies on the question order effect conducted in other STEM fields.