Audiovisual emotion recognition based on a multi-head cross attention mechanism.
10.7507/1001-5515.202406041
- Author:
Ziqiong WANG
1
;
Dechun ZHAO
1
;
Lu QIN
1
;
Yi CHEN
1
;
Yuchen SHEN
1
Author Information
1. School of Life Health Information Science and Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, P. R. China.
- Publication Type:Journal Article
- Keywords:
Cross-attention;
Emotional analysis;
Modal fusion;
Representational learning
- MeSH:
Emotions;
Humans;
Attention;
Algorithms
- From:
Journal of Biomedical Engineering
2025;42(1):24-31
- CountryChina
- Language:Chinese
-
Abstract:
In audiovisual emotion recognition, representational learning is a research direction receiving considerable attention, and the key lies in constructing effective affective representations with both consistency and variability. However, there are still many challenges to accurately realize affective representations. For this reason, in this paper we proposed a cross-modal audiovisual recognition model based on a multi-head cross-attention mechanism. The model achieved fused feature and modality alignment through a multi-head cross-attention architecture, and adopted a segmented training strategy to cope with the modality missing problem. In addition, a unimodal auxiliary loss task was designed and shared parameters were used in order to preserve the independent information of each modality. Ultimately, the model achieved macro and micro F1 scores of 84.5% and 88.2%, respectively, on the crowdsourced annotated multimodal emotion dataset of actor performances (CREMA-D). The model in this paper can effectively capture intra- and inter-modal feature representations of audio and video modalities, and successfully solves the unity problem of the unimodal and multimodal emotion recognition frameworks, which provides a brand-new solution to the audiovisual emotion recognition.
