Off-campus UNL users: To download campus access dissertations, please use the following link to log into our proxy server with your NU ID and password. When you are done browsing please remember to return to this page and log out.

Non-UNL users: Please talk to your librarian about requesting this dissertation through interlibrary loan.

Using Recurrent Neural Networks to Code Interviewer Question-Asking Behaviors: A Proof of Concept

Jerry Timbrook, University of Nebraska - Lincoln


Survey researchers use behavior coding to identify when interviewers do not read survey questions exactly as worded in the questionnaire, creating a potential source of interviewer variance. Manual behavior coding of an entire survey, however, can be expensive and time-consuming. In this dissertation, I examine whether this process can be partially automated by Recurrent Neural Networks (RNNs; a machine learning technique that can learn to categorize sequential data using patterns learned from previously categorized examples), saving time and money. Humans use transcripts from 33 questions in the Work and Leisure Today II telephone survey (WLT2) to manually behavior code n=25,442 interviewer question-askings. Using these transcripts as learning examples, I train RNNs to classify interviewer question-asking behaviors. I also code these transcripts using two string-based methods: 1) Levenshtein distance (i.e., the number of characters in the question-asking transcript that need to be changed to match the questionnaire text exactly); and 2) exact string matching (i.e., whether the question-asking transcript matches the questionnaire text exactly). A random 10% subsample of transcripts (n=2,650) was also coded by master coders to evaluate inter-coder reliability. I compare the reliability of RNN coding versus the master coders to the reliability of human and string-based coding. I also evaluate generalizability by using these RNNs to code n=4,711 question-asking transcripts from 13 of the same questions in a different telephone survey fielded at a different time, Work and Leisure Today I (WLT1). Results demonstrate that RNN coding of interviewer question-asking behaviors was comparable to human coding for most questions in this dissertation. The coding utility of Levenshtein distance and exact matching were limited relative to the RNNs and human coders. Finally, the RNNs had comparable reliability when coding questions from both WLT1 and WLT2, but only when question-asking behaviors were similar across the two surveys. Overall, this dissertation provides a proof-of-concept for using RNNs to code interviewer question-asking behaviors, but more research is necessary to refine the settings used to train these networks.

Subject Area

Sociology|Artificial intelligence|Behavioral Sciences

Recommended Citation

Timbrook, Jerry, "Using Recurrent Neural Networks to Code Interviewer Question-Asking Behaviors: A Proof of Concept" (2020). ETD collection for University of Nebraska - Lincoln. AAI28085989.