Comparing survey ranking question formats in mail surveys

Authors

Jolene Smyth, University of Nebraska-LincolnFollow
Kristen Olson, University of Nebraska-LincolnFollow
Allison Burke, LPC Consulting Associates, Inc.

Document Type

Article

Date of this Version

2018

Citation

Published in International Journal of Market Research (2018), 15pp.

doi 10.1177/1470785318767286

Comments

Copyright © 2018 Jolene D. Smyth, Kristen Olson, and Allison Burke. Published by SAGE Publications. Used by permission.

Abstract

Although questions that ask respondents to rank-order a list of items can be analytically valuable, responding to ranking questions typically requires a good deal of cognitive effort. This is especially true in mail questionnaires where the advantages of electronic response formats available in web surveys are inaccessible. In this article, we examine two alternative formats for ranking questions in mail surveys. Using a nationally representative mail survey of U.S. adults, this article experimentally compares ranking formats in which respondents write numbers in boxes versus selecting items for the most and second most important issues using a grid layout. Respondents to the numbering format were more likely to provide usable data, although one-third of respondents in this format still did not follow instructions correctly. Substantive responses differed somewhat across formats. Less educated respondents had difficulty with both formats, resulting in substantively different conclusions about preferences across formats for this group. A numbering format is more effective than a most–second most grid format for collecting ranking data in mail surveys.