Sociology, Department of
Date of this Version
2-26-2019
Document Type
Article
Citation
Presented at “Interviewers and Their Effects from a Total Survey Error Perspective Workshop,” University of Nebraska-Lincoln, February 26-28, 2019.
Abstract
Interviewer effects remain a pervasive problem in survey research. Utilizing a mixed-methods approach, this study explores the effects that interviewers have on the reporting of agricultural workers on the USDA’s Agricultural Labor Survey. The Agricultural Labor Survey is administered by the USDA’s National Agricultural Statistics Service on a biannual basis, with each data collection period collecting information for two quarters (e.g. April and January of 2018). While the majority of data is collected via computer-assisted telephone interviewing, a sizeable proportion is completed online or via mail. Using quantitative and qualitative methods, this study explores the patterns of bias introduced by interviewers, as well as some of the mechanisms by which these biases are introduced. Inverse probability weighting (IPW) is used to examine the variation in the number and type of agricultural workers reported across different modes of survey administration. Preliminary analyses show that the mode of administration is associated with the number and types of agricultural workers reported on the Agricultural Labor Survey, after adjusting for respondents’ propensity to complete the survey using these modes of administration. In addition, behavior coding of 39 CATI interviews conducted in April of 2018 was completed to examine how CATI interviewers administer the Agricultural Labor Survey. The behavior coding results show that interviewers frequently alter the meaning of critical questions related to the reporting and categorization of agricultural workers, or fail to ask them altogether. For example, less than a quarter of interviewers read the required text designed to expose respondents to the four major agricultural worker categories of interest (field, livestock, supervisor, and other workers). Even fewer interviewers correctly administered the questions exposing respondents to the worker subcategories associated with each of the four major worker categories, a pattern that is exacerbated when collecting information for workers hired in January of 2018. Examining the administration of these survey questions in conjunction with quantitative analyses is crucial as it sheds light on the manner in which interviewers potentially affect the reporting of agricultural workers. Altogether, using a mixed-methods approach to uncover interviewer effects is an effective strategy for both identifying the bias introduced by interviewers as well as the ways by which they contribute to the total survey error. Potential applications of this approach and the implications of these findings for the training and development of CATI interviewers within an operational setting are discussed.
Comments
Copyright 2019 by the authors.