Sociology, Department of
Date of this Version
2-26-2019
Document Type
Article
Citation
Presented at “Interviewers and Their Effects from a Total Survey Error Perspective Workshop,” University of Nebraska-Lincoln, February 26-28, 2019.
Abstract
Interviewers play a critical role in determining the quality of data collected in face-to-face surveys. Interviewers can have positive effects on recruiting sample members to participate, leading to higher response rates. Conversely, interviewers can have negative effects on the quality of measurement. The literature suggests that interviewers can bias answers when observable characteristics of the interviewer influence the respondent to answer questions a certain way. For example, the sex or race of interviewers may influence respondents’ answers about their own attitudes toward sex or race. However, it is more common for differences in interviewer behavior, such as how questions are asked or how answers are probed, to affect the variability of responses. These differences in interviewer characteristics and behaviors lead to answers that are clustered by the interviewer giving rise to a within interviewer correlation that inflates the estimated variability of survey statistics. The size of this increased variability or interviewer effect is often difficult to estimate in face-to-face surveys since standard estimation techniques assume interpenetrated designs that randomly assign interviewers to areas. Instead, multilevel models that control for respondent and area effects are often used to isolate interviewer effects from area effects in non-interpenetrated designs.
This study uses multilevel models to model interviewer effects in the National Health Interview Survey (NHIS), a large national survey of approximately 35,000 households conducted annually. The NHIS is an entirely interviewer-administered survey conducted primarily face-to-face with some telephone follow-up. Using 2017 data, we begin by estimating multilevel models to compute estimates of interviewer variance across a variety of questions in the NHIS. The goal is to determine the extent to which interviewer variance is present in NHIS estimates. The analysis will include questions that vary by characteristics such as question sensitivity, question length, and response format. The models will include controls for Census demographics within areas to help separate interviewer effects from area effects. The next step in the analysis will attempt to understand the extent to which certain interviewer-level variables can explain the interviewer effects, including how much of the interviewer-level variance is explained by interviewer experience. We also include a measure of the interviewers’ cooperation rates to understand if differences in nonresponse error may explain some of the interviewer-level variance in key survey estimates. Finally, we will include interviewer-level variables such as average pace of the interview to understand how much of the variance may be explained by interviewer behavior. The overall goals of the paper are to 1) understand which questions on the NHIS are most vulnerable to interviewer effects, and 2) explain the relative impact of different potential causes of those effects.
Comments
Copyright 2019 by the authors.