Sociology, Department of
Date of this Version
2-26-2019
Document Type
Article
Citation
Presented at “Interviewers and Their Effects from a Total Survey Error Perspective Workshop,” University of Nebraska-Lincoln, February 26-28, 2019.
Abstract
Undesirable interviewer behavior (UIB) could be one source for data errors and measurement effects in the setting of standardized interviewing techniques. Survey organizations have to ensure that errors and effects are minimized by validating their data collection processes during the entire survey period.
Monitoring is one method of validation which has been well established for telephone surveys from their very beginning. Moreover, it is one of the advantage of telephone interviews compared to face-to-face interviews. In most survey organizations it includes listening to interviews at the time they are being carried out by either supervisors or clients resp. scientists.
For face-to-face interviews live monitoring would only be possible by accompanying the interviewers visiting their respondents. Such a companion means a lot more effort in costs and organizational procedures. However, this may also have an impact on the data collection in as much as a third person can influence the interview situation and thus the respective data quality. A much more efficient way of monitoring face-to-face interviews is listening to recordings of the interviews afterwards. Audio recordings can be produced easily due to face-to-face interviews being carried out as CAPI so that they may be recorded and stored as digital sound file on the computer. Since no additional recording device such as a tape recorder is visible, recording is less obtrusive and respondents and interviewers are more likely to forget about it during the interview. Regrettably, this procedure can result in refusals because the respondents need to consent to efforts necessary due to listening and coding afterwards. How can we handle audio files in large scale surveys? Is it possible to install a monitoring process based on audio recordings for face-to-face interviews?
For the PASS Panel Study “Labour Market and Social Security” we established at infas a procedure within the last three years for monitoring face-to-face interviews (CAPI). The PASS panel started in 2006 and has since then been running annually with approximately 12,000 persons in more than 8,000 households.
With its mixed-mode design about two thirds of the interviews are conducted as CAPI. Every face-to-face interview shall be recorded which results in over 5,7000 audio recordings e.g., in wave 12. We found high acceptance for recording without risking refusals. A small amount of audio files is executed to detect UIB at an early stage of fieldwork. Coding is limited on main issues of standardized interviewing e.g. inadequate readings of questions or probing behavior. In case of UIB interviewers get feedback or will be trained again.
With this paper we like to describe our monitoring procedure and experiences with coding and feedback routines for face-to-face interviews starting right at the beginning of the fieldwork period. Our database also allows for some evaluation in terms of efficiency and effort for monitoring, detecting of deviations and improving interviewers’ behavior.
Comments
Copyright 2019 by the authors.