Self-attention fusion for audiovisual emotion recognition with incomplete data
TL
TL;DR
This paper proposes an architecture capable of learning from raw data and describes three variants of it with distinct modality fusion mechanisms that improves performance drastically under the absence/noisy representations of one modality, and improves the performance in a standard ideal setting, outperforming the competing methods.
Abstract
In this paper, we consider the problem of multi-modal data analysis with a use case of audiovisual emotion recognition. We propose an architecture capable of learning from raw data and describe three variants of it with distinct modality fusion mechanisms. While most of the previous works consider the ideal scenario of presence of both modalities at all times during inference, we evaluate the robustness of the model in the unconstrained settings where one modality is absent or noisy, and propose a method to mitigate these limitations in a form of modality dropout. Most importantly, we find that following this approach not only improves performance drastically under the absence/noisy representations of one modality, but also improves the performance in a standard ideal setting, outperforming the competing methods.
References24 items
1
Learning to ignore: rethinking attention in CNNs
2
Robust Lightweight Facial Expression Recognition Network with Label Distribution Training
3
Deep Multimodal Fusion by Channel Exchanging
4
Multimodal Emotion Recognition Using Cross-Modal Attention and 1D Convolutional Neural Networks
5