A Multimodal Approach for Building Generalizable and Reliable Models of Longitudinal Experiential Data

Authors

• Ahatsham Hayat, University of Nebraska-LincolnFollow

First Advisor

Mohammad Rashedul Hasan

Degree Name

Doctor of Philosophy (Ph.D.)

Committee Members

Benjamin Riggan, Hamid Sharif-Kashani, Siamak Nejati

Department

Computer Engineering

Date of this Version

4-27-2026

Document Type

Dissertation

Citation

A dissertation presented to the faculty of the Graduate College at the University of Nebraska in partial fulfillment of requirements for the degree Doctor of Philosophy

Major: Computer Engineering

Under the supervision of Professor Mohammad Rashedul Hasan

Lincoln, Nebraska, May 2026

Comments

Copyright 2026, Ahatsham Hayat. Used by permission

Abstract

This dissertation studies how to build generalizable and reliable models for longitudinal experiential (LE) data. Such data arises in domains where human behavior, experience, and context evolve over time, including education, behavioral health, and related human-centered settings. These datasets are often heterogeneous, partially observed, temporally structured, and vulnerable to out-of-distribution (OOD) shift, making them difficult to model with conventional machine learning pipelines.

The dissertation develops a multimodal research program that progresses from traditional machine learning and deep learning methods to contextual large language modeling, vision-language modeling (VLM), and finally a frozen-backbone multimodal framework designed to improve robustness and generalization. Across this progression, the dissertation shows that effective LE modeling depends critically on representation design, contextual reasoning, missingness-aware learning, structure-preserving multimodal encoding, and training constraints that reduce source-specific shortcut learning.

In this dissertation, reliability means more than achieving high in-distribution accuracy. It means producing predictions that remain stable, semantically coherent, and useful when data are incomplete and when the test distribution differs from the training distribution. The results show a clear progression in OOD performance across the three main benchmarks. On GLOBEM, the strongest OOD accuracy improves from 51.06% with traditional baselines to 67.40% with text-only language modeling, 72.86% with multimodal vision-language modeling, and 79.93% with the final constrained frozen-backbone framework. Similar gains appear on LifeSnaps, where performance rises from 48.44% to 67.19%, 71.88%, and 81.25%, and on MFAFY, where it improves from 50.57% to 64.86%, 66.57%, and 73.71%. Together, these findings show that reliable LE modeling emerges when semantic context, temporal structure, and complementary modalities are aligned with frozen priors that preserve transfer-relevant behavior under distribution shift.

Recommended Citation

Hayat, Ahatsham, "A Multimodal Approach for Building Generalizable and Reliable Models of Longitudinal Experiential Data" (2026). Dissertations and Doctoral Documents, University of Nebraska-Lincoln, 2023–. 466.
https://digitalcommons.unl.edu/dissunl/466