Self-Supervised Audio-Visual Representation Learning with Relaxed Cross-Modal Synchronicity (2021-11-09T00:00:00.000000Z)

TL;DR

The experiments show that CrissCross either outperforms or achieves performances on par with the current state-of-the-art self-supervised methods on action recognition and action retrieval with UCF101 and HMDB51, as well as sound classification with ESC50 and DCASE.

Abstract

We present CrissCross, a self-supervised framework for learning audio-visual representations. A novel notion is introduced in our framework whereby in addition to learning the intra-modal and standard 'synchronous' cross-modal relations, CrissCross also learns 'asynchronous' cross-modal relationships. We perform in-depth studies showing that by relaxing the temporal synchronicity between the audio and visual modalities, the network learns strong generalized representations useful for a variety of downstream tasks. To pretrain our proposed solution, we use 3 different datasets with varying sizes, Kinetics-Sound, Kinetics400, and AudioSet. The learned representations are evaluated on a number of downstream tasks namely action recognition, sound classification, and action retrieval. Our experiments show that CrissCross either outperforms or achieves performances on par with the current state-of-the-art self-supervised methods on action recognition and action retrieval with UCF101 and HMDB51, as well as sound classification with ESC50 and DCASE. Moreover, CrissCross outperforms fully-supervised pretraining while pretrained on Kinetics-Sound.

Authors

Pritam Sarkar

2 papers

Ali Etemad

2 papers

TL;DR

Abstract

Authors

References80 items

Self-supervised Contrastive Learning of Multi-view Facial Expressions

Cross-Modal Attention Consistency for Video-Audio Unsupervised Learning

VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

Broaden Your Views for Self-Supervised Video Learning

Robust Audio-Visual Instance Discrimination

Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning

BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation

Exploring Simple Siamese Representation Learning

Self-Supervised Learning with Cross-Modal Transformers for Emotion Recognition

Detection of Maternal and Fetal Stress from ECG with Self-supervised Representation Learning

Self-supervised Co-training for Video Representation Learning

Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion

Self-Supervised Video Representation Learning by Uncovering Spatio-Temporal Statistics

Spatiotemporal Contrastive Video Representation Learning

Self-Supervised MultiModal Versatile Networks

Labelling unlabelled videos from scratch with multi-modal self-supervision

Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Learning To Classify Images Without Labels

Audio-Visual Instance Discrimination with Cross-Modal Agreement

SpeedNet: Learning the Speediness in Videos

On Compositions of Transformations in Contrastive Self-Supervised Learning

Self-Supervised Spatio-Temporal Representation Learning Using Variable Playback Speed Prediction

Evolving Losses for Unsupervised Video Representation Learning

A Simple Framework for Contrastive Learning of Visual Representations

Self-Supervised ECG Representation Learning for Emotion Recognition

Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning

Self-Supervised Learning of Pretext-Invariant Representations

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Self-Supervised Learning by Cross-Modal Audio-Video Clustering

Self-Supervised Learning for ECG-Based Emotion Recognition

Learning Video Representations using Contrastive Bidirectional Transformer

Video Representation Learning by Dense Predictive Coding

Smooth Grad-CAM++: An Enhanced Inference Level Visualization Technique for Deep Convolutional Neural Network Models

Fixing the train-test resolution discrepancy

Self-Supervised Spatiotemporal Learning via Video Clip Order Prediction

Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition

SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition

Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction.

Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles

SDC-Net: Video Prediction Using Spatially-Displaced Convolution

Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition

Improving Spatiotemporal Self-Supervision by Deep Reinforcement Learning

Deep Clustering for Unsupervised Learning of Visual Features

Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization

A Closer Look at Spatiotemporal Convolutions for Action Recognition

Stochastic Variational Video Prediction

Mixed Precision Training

Improved Regularization of Convolutional Neural Networks with Cutout

Large Batch Training of Convolutional Networks

Unsupervised Representation Learning by Sorting Sequences

Dual Motion GAN for Future-Flow Embedded Video Prediction

MoCoGAN: Decomposing Motion and Content for Video Generation

Look, Listen and Learn

The Kinetics Human Action Video Dataset

Audio Set: An ontology and human-labeled dataset for audio events

Temporal Generative Adversarial Nets with Singular Value Clipping

Generating Videos with Scene Dynamics

SGDR: Stochastic Gradient Descent with Warm Restarts

Unsupervised Learning for Physical Interaction through Video Prediction

Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification

Deep Residual Learning for Image Recognition

Deep End2End Voxel2Voxel Prediction

Deep multi-scale video prediction beyond mean square error

ESC: Dataset for Environmental Sound Classification

Detection and Classification of Acoustic Scenes and Events

Adam: A Method for Stochastic Optimization

UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

HMDB: A large video database for human motion recognition

Signature Verification Using A "Siamese" Time Delay Neural Network

Active Contrastive Learning of Audio-Visual Video Representations

2021). Finally, we use the learned visual representations to train a linear SVM classifier, we sweep the cost values between

Following the protocols mentioned in (Alayrac et al

2021a), we extract 10 epochs worth

2019) to perform Time Mask, Frequency Mask, and Time Warp transformations2. The parameters for the audio augmentations are set empirically, except for Random Crop which we adopt from (Niizumi

Pretraining Datasets We use 3 datasets of different sizes for pretraining, namely, Kinetics-Sound (Arandjelovic and Zisserman 2017)