Optimization of Multimodal Deep Learning for Depression Detection

Abstract

Depression is a complex and often underdiagnosed mental health condition that manifests through subtle verbal, acoustic, and behavioral cues. Traditional unimodal detection systems struggle to capture the full spectrum of depressive symptoms, often leading to inaccurate or incomplete assessments. This study proposes a multimodal deep learning framework that integrates textual, audio, and visual modalities to improve the robustness and reliability of automatic depression detection, achieving an overall classification accuracy of 74%. The approach prioritizes privacy and interpretability by using facial keypoints and gaze direction rather than raw video frames, and applies attention mechanisms to align and fuse features across modalities. Each modality is processed through dedicated neural architectures tailored to its data type, and their outputs are combined within a fusion model that learns to capture cross-modal emotional patterns. Experimental results demonstrate that the proposed multimodal system significantly outperforms its unimodal counterparts in terms of classification performance. The visual modality was found to contribute most strongly to detection accuracy, as confirmed by ablation analysis. These findings highlight the value of multimodal integration in capturing complex psychological signals and support the development of intelligent, non-invasive screening tools for use in digital mental health applications.