Complementary Fusion of Multi-Features and Multi-Modalities in Sentiment Analysis (2019-04-17T00:00:00.000000Z)

TL;DR

Surprisingly, DFF-ATMF also achieves new state-of-the-art results on the IEMOCAP dataset, indicating that the proposed fusion strategy also has a good generalization ability for multimodal emotion recognition.

Abstract

Sentiment analysis, mostly based on text, has been rapidly developing in the last decade and has attracted widespread attention in both academia and industry. However, the information in the real world usually comes from multiple modalities, such as audio and text. Therefore, in this paper, based on audio and text, we consider the task of multimodal sentiment analysis and propose a novel fusion strategy including both multi-feature fusion and multi-modality fusion to improve the accuracy of audio-text sentiment analysis. We call it the DFF-ATMF (Deep Feature Fusion - Audio and Text Modality Fusion) model, which consists of two parallel branches, the audio modality based branch and the text modality based branch. Its core mechanisms are the fusion of multiple feature vectors and multiple modality attention. Experiments on the CMU-MOSI dataset and the recently released CMU-MOSEI dataset, both collected from YouTube for sentiment analysis, show the very competitive results of our DFF-ATMF model. Furthermore, by virtue of attention weight distribution heatmaps, we also demonstrate the deep features learned by using DFF-ATMF are complementary to each other and robust. Surprisingly, DFF-ATMF also achieves new state-of-the-art results on the IEMOCAP dataset, indicating that the proposed fusion strategy also has a good generalization ability for multimodal emotion recognition.

Authors

Feiyang Chen

4 papers

Ziqian Luo

1 papers

Yanyan Xu

1 papers

TL;DR

Abstract

Authors

References35 items

Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Audio Sentiment Analysis by Heterogeneous Signal Features Learned from Utterance-Based Parallel Neural Network

Distinguishing between facts and opinions for sentiment analysis: Survey and challenges

Learning and Fusing Multimodal Deep Features for Acoustic Scene Categorization

Multimodal Speech Emotion Recognition Using Audio and Text

Transformer-XL: Language Modeling with Longer-Term Dependency

Convolutional Neural Network Techniques for Speech Emotion Recognition

Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling

THU_NGN at SemEval-2018 Task 1: Fine-grained Tweet Sentiment Intensity Analysis with Attention CNN-LSTM

Multi-view and Attention-Based BI-LSTM for Weibo Emotion Recognition

Convolutional Attention Networks for Multimodal Emotion Recognition from Speech and Text Data

Speech emotion recognition

Multimodal Sentiment Analysis: Addressing Key Issues and Setting Up the Baselines

Evolving netnography: how brand auto-netnography, a netnographic sensibility, and more-than-human netnography can transform your research

Deep learning for sentiment analysis: A survey

Speech emotion recognition based on feature selection and extreme learning machine decision tree

Multi-level Multiple Attentions for Contextual Multimodal Sentiment Analysis

A review of affective computing: From unimodal analysis to multimodal fusion

Context-Dependent Sentiment Analysis in User-Generated Videos

Multimodal Machine Learning: A Survey and Taxonomy

A first look into a Convolutional Neural Network for speech emotion detection

MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos

A survey on sentiment analysis challenges

Effective Approaches to Attention-based Neural Machine Translation

Speech emotion recognition using deep neural network and extreme learning machine

Emotion recognition using a hierarchical binary decision tree approach

Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine-belief network architecture

Hidden Markov model-based speech emotion recognition

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Combining Textual Clues with Audio-Visual Information for Multimodal Sentiment Analysis

Contextual Inter-modal Attention for Multi-modal Sentiment Analysis

Deep Convolution Neural Networks for Twitter Sentiment Analysis

librosa: Audio and Music Signal Analysis in Python

Field of Study