Convolutional neural networks can decode eye movement data: A black box approach to predicting task from eye movements

Authors

Zachary J. Cole, University of Nebraska-LincolnFollow
Karl M. Kuntzelman, University of Nebraska-LincolnFollow
Michael D. Dodd, University of Nebraska‐LincolnFollow
Matthew R. Johnson, University of Nebraska–LincolnFollow

Document Type

Article

Date of this Version

2021

Citation

Cole, Z. J., Kuntzelman, K. M., Dodd, M. D., & Johnson, M. R. (2021). Convolutional neural networks can decode eye movement data: A black box approach to predicting task from eye movements. Journal of Vision, 21(7):9, 1–15, https://doi.org/10.1167/jov.21.7.9.

Comments

ISSN 1534-7362 Copyright 2021 The Authors This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Abstract

Previous attempts to classify task from eye movement data have relied on model architectures designed to emulate theoretically defined cognitive processes and/or data that have been processed into aggregate (e.g., fixations, saccades) or statistical (e.g., fixation density) features. Black box convolutional neural networks (CNNs) are capable of identifying relevant features in raw and minimally processed data and images, but difficulty interpreting these model architectures has contributed to challenges in generalizing lab-trained CNNs to applied contexts. In the current study, a CNN classifier was used to classify task from two eye movement datasets (Exploratory and Confirmatory) in which participants searched, memorized, or rated indoor and outdoor scene images. The Exploratory dataset was used to tune the hyperparameters of the model, and the resulting model architecture was retrained, validated, and tested on the Confirmatory dataset. The data were formatted into timelines (i.e., x-coordinate, y-coordinate, pupil size) and minimally processed images. To further understand the informational value of each component of the eye movement data, the timeline and image datasets were broken down into subsets with one or more components systematically removed. Classification of the timeline data consistently outperformed the image data. The Memorize condition was most often confused with Search and Rate. Pupil size was the least uniquely informative component when compared with the x- and y-coordinates. The general pattern of results for the Exploratory dataset was replicated in the Confirmatory dataset. Overall, the present study provides a practical and reliable black box solution to classifying task from eye movement data.