Sociology, Department of
Date of this Version
2-26-2019
Document Type
Article
Citation
Presented at “Interviewers and Their Effects from a Total Survey Error Perspective Workshop,” University of Nebraska-Lincoln, February 26-28, 2019.
Abstract
Within survey methodology it is common knowledge that interviewers in face-to-face or telephone interviews can have undesirable effects on the obtained answers. These effects can be created in an active way by, for example, asking suggestive questions or they can be obtained in a passive way as a consequence of certain interviewer characteristics eliciting socially desirable answers. These active and passive effects may differ from interviewer to interviewer. These differences between interviewers in systematic effects create additional variance in the data. The proportion of variance in a (substantive) variable that can be explained by the interviewers is the ‘so called’ between-interviewer variance. It is clear that high between-interviewer variance results in a negative assessment of data quality. Notice that not all types of interviewer effects (e.g. ‘pure’ interviewer bias) can be evaluated by means of the analysis of interviewer variance.
A frequently used measure for the evaluation of interviewer variance is the intra class correlation coefficient (ICC). This coefficient expresses the homogeneity of the obtained answers within the interviewers compared with the heterogeneity of the answers between the interviewers. To calculate the within and the between variance components it is important to take into account the two-level hierarchical data structure in which respondents are nested within the interviewers. A two-level random intercept model with no independent variables is generally the starting point for such an analysis. The model provides estimates of the within and between interviewer variance used to calculate the basic value of the ICC. The basic model can be elaborated by interviewer characteristics (e.g. experience, workload, gender, ...) at the interviewer level and respondent characteristics at the respondent level. With the interviewer characteristics one can try to explain the between interviewer variance. If these characteristics partly explain this variance, they give some insight into the mechanism behind the interviewer's effects.
In contrast, the evaluation of the impact of respondent characteristics (and characteristics of the interview situation) specified at the respondent level is less obvious. With respondent characteristics one tries to explain the variance of the substantive dependent variable of the model. But in the context of the evaluation of interviewer variance, respondent characteristics are also specified in the model to control for differences between interviewers in the composition of the respondent groups. The impact of the interviewers are evaluated after respondent characteristics explained part of the variance in the dependent variable. This means that respondent characteristics are used to explain the variance in the substantive dependent variable and that interviewer effects express the variability between interviewers after controlling for these respondent characteristics. Such models do not assess the effect of respondent characteristics on interviewer effects. In fact, the relationship between the respondent characteristics and the interviewer effects is not specified in the model. However it is reasonable to assume that some respondents are more sensitive to interviewer effects and that in some respondent groups the ICCs are higher. So the specification of the basic multi-level model does not really allow to investigate the relationship between certain respondent characteristics and the extent to which these characteristics influence interviewer effects. In this paper, various alternative specifications of the basic model in which this relationship is explicitly specified will be explored and compared with each other.
Comments
Copyright 2019 by the authors.