Visible-to-Thermal Transfer Learning for Facial Landmark Detection

Authors

Domenick D. Poster, West Virginia University
Shuowen Hu, U.S. Army Combat Capabilities Development Command (DEVCOM) Army Research Laboratory
Nathan J. Short, Booz Allen Hamilton, Laurel, MD
Benjamin S. Riggan, University of Nebraska - LincolnFollow
Nasser M. Nasrabadi, West Virginia University

ORCID IDs

Domenick D. Poster

Benjamin S. Riggan

Nasser M. Nasrabadi

Document Type

Article

Date of this Version

4-12-2021

Citation

Digital Object Identifier 10.1109/ACCESS.2021.3070233

Comments

CC-BY

Abstract

There has been increasing interest in face recognition in the thermal infrared spectrum. A critical step in this process is face landmark detection. However, landmark detection in the thermal spectrum presents a unique set of challenges compared to in the visible spectrum: inherently lower spatial resolution due to longer wavelength, differences in phenomenology, and limited availability of labeled thermal face imagery for algorithm development and training. Thermal infrared imaging does have the advantage of being able to passively acquire facial heat signatures without the need for active or ambient illumination in low light and nighttime environments. In such scenarios, thermal imaging must operate by itself without corresponding/paired visible imagery. Mindful of this constraint, we propose visible-to-thermal parameter transfer learning using a coupled convolutional network architecture as a means to leverage visible face data when training a model for thermal-only face landmark detection. This differentiates our approach from models trained either solely on thermal images or models which require a fusion of visible and thermal images at test time. In this work, we implement and analyze four types of parameter transfer learning methods in the context of thermal face landmark detection: Siamese (shared) layers, Linear Layer Regularization (LLR), Linear Kernel Regularization (LKR), and Residual Parameter Transformations (RPT). These transfer learning approaches are compared against a baseline version of the network and an Active Appearance Model (AAM), both of which are trained only on thermal data. We achieve a 6.5% - 9.5% improvement on the DEVCOM ARL Multi-modal Thermal Face Dataset and a 4% improvement on the RWTH Aachen University Thermal Face Dataset over the baseline model. We show that LLR, LKR, and RPT all result in improved thermal face landmark detection performance compared to the baseline and AAM, demonstrating that transfer learning leveraging visible spectrum data improves thermal face landmarking.