Contrastive Audio-Visual Masked Autoencoder

Published in

International Conference on Learning Representa...(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

The Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) is proposed by combining contrastive learning and masked data modeling, two major self-supervised learning frameworks, to learn a joint and coordinated audio-visual representation.

Abstract

In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities. Subsequently, we propose the Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) by combining contrastive learning and masked data modeling, two major self-supervised learning frameworks, to learn a joint and coordinated audio-visual representation. Our experiments show that the contrastive audio-visual correspondence learning objective not only enables the model to perform audio-visual retrieval tasks, but also helps the model learn a better joint representation. As a result, our fully self-supervised pretrained CAV-MAE achieves a new SOTA accuracy of 65.9% on VGGSound, and is comparable with the previous best supervised pretrained model on AudioSet in the audio-visual event classification task. Code and pretrained models are at https://github.com/yuangongnd/cav-mae.

Authors

Hilde Kuehne

5 papers

Yuan Gong

7 papers

James R. Glass

16 papers

References60 items

Multi-modal Deep Learning

Masked Vision and Language Modeling for Multi-modal Representation Learning

Contrastive Masked Autoencoders are Stronger Vision Learners

Masked Autoencoders that Listen

OmniMAE: Single Model Masked Pretraining on Images and Videos

Contrastive Audio-Visual Masked Autoencoder

Published in

International Conference on Learning Representa...(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Hilde Kuehne

5 papers

Yuan Gong

7 papers

James R. Glass

16 papers

References60 items

Multi-modal Deep Learning

Masked Vision and Language Modeling for Multi-modal Representation Learning

Contrastive Masked Autoencoders are Stronger Vision Learners

Masked Autoencoders that Listen

OmniMAE: Single Model Masked Pretraining on Images and Videos

Leonid Karlinsky

2 papers

David F. Harwath

3 papers

Alexander H. Liu

3 papers

Andrew Rouditchenko

3 papers

Multimodal Masked Autoencoders Learn Transferable Representations

Masked Autoencoders As Spatiotemporal Learners

One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Code

Masked Spectrogram Prediction for Self-Supervised Audio Pre-Training

Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representation

MultiMAE: Multi-modal Multi-task Masked Autoencoders

MAE-AST: Masked Autoencoding Audio Spectrogram Transformer

AudioTagging Done Right: 2nd comparison of deep learning methods for environmental sound classification

VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training

HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection

Masked Feature Prediction for Self-Supervised Visual Pre-Training

Masked Autoencoders Are Scalable Vision Learners

SSAST: Self-Supervised Audio Spectrogram Transformer

Conformer-Based Self-Supervised Learning For Non-Speech Audio Tasks

Efficient Training of Audio Transformers with Patchout

Attention Bottlenecks for Multimodal Fusion

BEiT: BERT Pre-Training of Image Transformers

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

Multimodal Self-Supervised Learning of General Audio Representations

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

AST: Audio Spectrogram Transformer

Broaden Your Views for Self-Supervised Video Learning

Robust Audio-Visual Instance Discrimination

Slow-Fast Auditory Streams for Audio Recognition

Perceiver: General Perception with Iterative Attention

PSLA: Improving Audio Tagging With Pretraining, Sampling, Labeling, and Aggregation

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

AVLnet: Learning Audio-Visual Language Representations from Instructional Videos

Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data

Vggsound: A Large-Scale Audio-Visual Dataset

Audio-Visual Instance Discrimination with Cross-Modal Agreement

On Compositions of Transformations in Contrastive Self-Supervised Learning

Audiovisual SlowFast Networks for Video Recognition

PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition

What Makes Training Multi-Modal Classification Networks Hard?

Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization

Audio-Visual Scene Analysis with Self-Supervised Multisensory Features

Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning

Objects that Sound

mixup: Beyond Empirical Risk Minimization

Attention is All you Need

Multimodal Machine Learning: A Survey and Taxonomy

Look, Listen and Learn

The Kinetics Human Action Video Dataset

Audio Set: An ontology and human-labeled dataset for audio events

SoundNet: Learning Sound Representations from Unlabeled Video

MSR-VTT: A Large Video Description Dataset for Bridging Video and Language

Multimodal Deep Learning

Extracting and composing robust features with denoising autoencoders

Active Contrastive Learning of Audio-Visual Video Representations

Contrastive Learning of Global and Local Video Representations

20K is a subset of AudioSet-2M with more balanced class distribution. Due to changes in video availability, we downloaded 1.77M AudioSet-2M training, 19K AudioSet-20K training, and 17K evaluation

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Grounding Spoken Words in Unlabeled Video

Field of Study

Computer ScienceEngineering

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

International Conference on Learning Representations

Type

conference

URL

https://iclr.cc/

Alternate Names

Int Conf Learn Represent
ICLR