Comparing Readability Measures and Computer-assisted Question Evaluation Tools for Self-administered Survey Questions

Authors

Rachel Stenger, RTI International, Research Triangle Park, NCFollow
Kristen Olson, University of Nebraska-LincolnFollow
Jolene Smyth, University of Nebraska-LincolnFollow

ORCID IDs

Rachel Stenger https://orcid.org/0000-0003-0643-198X

Kristen Olson https://orcid.org/0000-0002-8004-0226

Document Type

Article

Date of this Version

2022

Citation

Published in Field Methods 2022

DOI: 10.1177/1525822X221124469

Comments

Copyright © 2022 Rachel Stenger, Kristen Olson, and Jolene D. Smyth; published by SAGE Publications. Used by permission.

Abstract

Questionnaire designers use readability measures to ensure that questions can be understood by the target population. The most common measure is the Flesch-Kincaid Grade level, but other formulas exist. This article compares six different readability measures across 150 questions in a self-administered questionnaire, finding notable variation in calculated readability across measures. Some question formats, including those that are part of a battery, require important decisions that have large effects on the estimated readability of survey items. Other question evaluation tools, such as the Question Understanding Aid (QUAID) and the Survey Quality Predictor (SQP), may identify similar problems in questions, making readability measures less useful. We find little overlap between QUAID, SQP, and the readability measures, and little differentiation in the tools’ prediction of item nonresponse rates. Questionnaire designers are encouraged to use multiple question evaluation tools and develop readability measures specifically for survey questions.