Sociology, Department of
Date of this Version
2-26-2019
Document Type
Article
Citation
Presented at “Interviewers and Their Effects from a Total Survey Error Perspective Workshop,” University of Nebraska-Lincoln, February 26-28, 2019.
Abstract
Consideration of interviewer effects (interviewer measurement error variance) in active quality control does not seem widespread despite its known effect on reducing precision of survey estimates. One major obstacle is that interviewer effect estimates computed on partial data (as a survey is in progress) can be very unstable. We address this issue by exploring the use of paradata (keystrokes and time stamps generated during the computer-assisted interviewing process) as proxies of interviewer effects with a focus on large-scale repeated cross-section or panel surveys.
We first estimate interviewer effects for each item in our analysis by using multilevel models that include a vector of respondent covariates to approximate interpenetration. We then compute the proportion of variance explained when we add interviewer-level paradata inputs to this model. These inputs are selected using adaptive lasso from a pool of thirteen measures. Realistic predictions of the explained variance are then computed using a bootstrap-based method.
We use data from the 2015 wave of the Panel Study of Income Dynamics (PSID) for our analysis and find promising results - paradata explain more than half the magnitude of interviewer effects on average across items. Also, paradata outperformed interviewer-level demographic and work-related variables in explaining interviewer effects. While most of the focus in the literature and practice has been on time-based paradata, e.g., item times, we find that non-time based paradata, e.g., frequency of item revisits, outperform the time-based paradata for a large majority of items. We conclude by discussing how survey organizations can use these findings in active quality control to contain interviewer effects.
Comments
Copyright 2019 by the authors.