Agreement answer scale design for multilingual surveys: Effects of translation-related changes in verbal labels on response styles and response distributions

Authors

Ana Villar, Stanford UniversityFollow

Date of this Version

2009

Document Type

Article

Comments

A DISSERTATION Presented to the Faculty of The Graduate College at the University of Nebraska In Partial Fulfillment of Requirements For The degree of Doctor of Philosophy, Major: Survey Research and Methodology, Under the Supervision of Professor Janet A. Harkness. Lincoln, Nebraska: December, 2009
Copyright 2009 Ana Villar.

Abstract

Answer scales in survey instruments are widely used, but little is known about how to choose verbal descriptors as labels. In multilingual research, this matter is further complicated because answer scales must be appropriate for all languages and function comparatively. Comparing source questionnaires to translations of multinational projects (e.g., the World Values Survey, the European Social Survey), it was observed that certain verbal features differed across languages, countries, and modules. This dissertation empirically investigates the effect of such changes on response distributions. The verbal feature examined is the presence or absence of an intensity modifier in the second and fourth labels of a 5-point agreement scale: strongly agree, (somewhat) agree, neither/nor, (somewhat) disagree, strongly disagree.

Two studies are conducted analyzing data from more than 40 countries of the International Social Survey Programme. The first investigates whether two methodological features (scale translation and administration mode) affect cross-cultural differences in three response styles. It was expected that adding an intensity modifier would increase extreme response style, decrease middle response style, and not affect acquiescence. Acquiescence was expected to be higher in interviewer-administered than in self-administered surveys. Using multilevel models in five ISSP modules, the analyses show that, as predicted, adding an intensity modifier results in higher extreme response style, and does not affect acquiescence. No support was found for the hypothesized effect on middle response style. Partial support was found for the hypothesized effect of data collection mode.

The second study investigates the effect of adding the intensity modifier on the use of each response category. Using an 8-item attitudinal scale, Graded Response Models for each of the answer scale versions were compared. For the answer scale under study, adding an intensity modifier made the scale points less useful to measure the underlying attitude than were scale points without modifier.

These findings suggest that modifications made to answer scale language versions are a critical source of variability in response patterns and distributions.