Audio-Visual Deception Detection: DOLOS Dataset and Parameter-Efficient Crossmodal Learning (2023-03-09T00:00:00.000000Z)

TL;DR

This work introduces DOLOS 1, the largest gameshow deception detection dataset with rich deceptive conversations, and proposes Parameter-Efficient Crossmodal Learning (PECL), where a Uniform Temporal Adapter (UT-Adapter) explores temporal attention in transformer-based architectures, and a crossmodal fusion module, Plug-in Audio-Visual Fusion (PAVF), combines crossmodal information from audio-visual features.

Abstract

Deception detection in conversations is a challenging yet important task, having pivotal applications in many fields such as credibility assessment in business, multimedia anti-frauds, and custom security. Despite this, deception detection research is hindered by the lack of high-quality deception datasets, as well as the difficulties of learning multimodal features effectively. To address this issue, we introduce DOLOS 1, the largest gameshow deception detection dataset with rich deceptive conversations. DOLOS includes 1, 675 video clips featuring 213 subjects, and it has been labeled with audio-visual feature annotations. We provide train-test, duration, and gender protocols to investigate the impact of different factors. We benchmark our dataset on previously proposed deception detection approaches. To further improve the performance by fine-tuning fewer parameters, we propose Parameter-Efficient Crossmodal Learning (PECL), where a Uniform Temporal Adapter (UT-Adapter) explores temporal attention in transformer-based architectures, and a crossmodal fusion module, Plug-in Audio-Visual Fusion (PAVF), combines crossmodal information from audio-visual features. Based on the rich fine-grained audio-visual annotations on DOLOS, we also exploit multi-task learning to enhance performance by concurrently predicting deception and audiovisual features. Experimental results demonstrate the desired quality of the DOLOS dataset and the effectiveness of the PECL. The DOLOS dataset and the source codes are available at here.

Authors

A. Kot

5 papers

Zitong Yu

9 papers

Xiaobao Guo

2 papers

TL;DR

Abstract

Authors

References55 items

LieNet: A Deep Convolution Neural Network Framework for Detecting Deception

Convolutional Bypasses Are Better Vision Transformer Adapters

Fine-Tuning Wav2Vec2 for Speaker Recognition

Highly accurate protein structure prediction with AlphaFold

Attention Bottlenecks for Multimodal Fusion

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

Deception Detection and Remote Physiological Monitoring: A Dataset and Baseline Experimental Results

MLP-Mixer: An all-MLP Architecture for Vision

Learning Transferable Visual Models From Natural Language Supervision

UniT: Multimodal Multitask Learning with a Unified Transformer

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Multimodal Political Deception Detection

Deception Detection in Videos Using Robust Facial Features

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Introducing Representations of Facial Affect in Automated Multimodal Deception Detection

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Language Models are Few-Shot Learners

Detection

Big Transfer (BiT): General Visual Representation Learning

PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition

Automatic Deception Detection in RGB videos using Facial Action Units

Box of Lies: Multimodal Deception Detection in Dialogues

Bag-of-Lies: A Multimodal Dataset for Deception Detection

Attention-augmented End-to-end Multi-task Learning for Emotion Prediction from Speech

Parameter-Efficient Transfer Learning for NLP

Face-Focused Cross-Stream Network for Deception Detection in Videos

A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play

Toward End-to-End Deception Detection in Videos

Miami University deception detection database

A Deep Learning Approach for Multimodal Deception Detection

Deception Detection in Videos

Deep learning driven multimodal fusion for automated deception detection

AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild

Attention is All you Need

Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks

Mastering the game of Go with deep neural networks and tree search

Deception Detection using Real-life Trial Data

Theorizing About Deception

Automatic deception detection in Italian court cases

Eliciting cues to deception and truth: What matters are the questions asked

Opensmile: the munich versatile and fast open-source audio feature extractor

Detecting Deception from Emotional and Unemotional Cues

Combining Prosodic Lexical and Cepstral Systems for Deceptive Speech Detection

Beyond kappa: A review of interrater agreement measures

Beyond the Pixel World: A Novel Acoustic-Based Face Anti-Spoofing System for Smartphones

Multimodal Deception Detection in Videos via Analyzing Emotional State-based Feature

End-to-End Audiovisual Speech Recognition System With Multitask Learning

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Improving Language Understanding by Generative Pre-Training

OpenFace: A general-purpose face recognition library with mobile applications

Verbal and Nonverbal Clues for Real-life Deception Detection

The MUMIN multimodal coding scheme

tinguishing deceptive from non - deceptive speech .

Cues to deception.

Ieee Transactions on Information Forensics and Security 1 Detecting Deceptive Behavior via Integration of Discriminative Features from Multiple Modalities

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names