Masked Autoencoders that Listen

Published in

Neural Information Processing Systems(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

The Audio-MAE, a simple extension of image-based Masked Autoencoders to self-supervised representation learning from audio spectrograms, sets new state-of-the-art performance on six audio and speech classification tasks, outperforming other recent models that use external supervised pre-training.

Abstract

This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers. The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram. We find it beneficial to incorporate local window attention in the decoder, as audio spectrograms are highly correlated in local time and frequency bands. We then fine-tune the encoder with a lower masking ratio on target datasets. Empirically, Audio-MAE sets new state-of-the-art performance on six audio and speech classification tasks, outperforming other recent models that use external supervised pre-training. The code and models will be at https://github.com/facebookresearch/AudioMAE.

Authors

Po-Yao (Bernie) Huang

5 papers

Michael Auli

14 papers

Wojciech Galuba

3 papers

References84 items

Masked Spectrogram Prediction for Self-Supervised Audio Pre-Training

Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representation

MAE-AST: Masked Autoencoding Audio Spectrogram Transformer

AudioTagging Done Right: 2nd comparison of deep learning methods for environmental sound classification

CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification

Masked Autoencoders that Listen

Published in

Neural Information Processing Systems(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Po-Yao (Bernie) Huang

5 papers

Michael Auli

14 papers

Wojciech Galuba

3 papers

References84 items

Masked Spectrogram Prediction for Self-Supervised Audio Pre-Training

Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representation

MAE-AST: Masked Autoencoding Audio Spectrogram Transformer

AudioTagging Done Right: 2nd comparison of deep learning methods for environmental sound classification

CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification

Christoph Feichtenhofer

9 papers

Florian Metze

5 papers

Alexei Baevski

6 papers

Juncheng Billy Li

1 papers

data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection

Robust Self-Supervised Audio-Visual Speech Recognition

Masked Feature Prediction for Self-Supervised Visual Pre-Training

MViTv2: Improved Multiscale Vision Transformers for Classification and Detection

Masked Autoencoders Are Scalable Vision Learners

SSAST: Self-Supervised Audio Spectrogram Transformer

Conformer-Based Self-Supervised Learning For Non-Speech Audio Tasks

Efficient Training of Audio Transformers with Patchout

Attention Bottlenecks for Multimodal Fusion

BEiT: BERT Pre-Training of Image Transformers

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

A Time-Domain Convolutional Recurrent Network for Packet Loss Concealment

ERANNs: Efficient Residual Audio Neural Networks for Audio Pattern Recognition

The Benefit of Temporally-Strong Labels in Audio Event Classification

SUPERB: Speech processing Universal PERformance Benchmark

AST: Audio Spectrogram Transformer

An Empirical Study of Training Self-Supervised Vision Transformers

Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning

Zero-Shot Text-to-Image Generation

PSLA: Improving Audio Tagging With Pretraining, Sampling, Labeling, and Aggregation

Training data-efficient image transformers & distillation through attention

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Generative Pretraining From Pixels

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Language Models are Few-Shot Learners

Voxceleb: Large-scale speaker verification in the wild

PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition

Deep Long Audio Inpainting

Momentum Contrast for Unsupervised Visual Representation Learning

Effectiveness of self-supervised pre-training for speech recognition

Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders

RoBERTa: A Robustly Optimized BERT Pretraining Approach

When Does Label Smoothing Help?

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition

A Context Encoder For Audio Inpainting

Representation Learning with Contrastive Predictive Coding

Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition

Objects that Sound

Decoupled Weight Decay Regularization

mixup: Beyond Empirical Risk Minimization

Attention is All you Need

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

A Deep Reinforced Model for Abstractive Summarization

Audio Set: An ontology and human-labeled dataset for audio events

Spatiotemporal Residual Networks for Video Action Recognition

CNN architectures for large-scale audio classification

SGDR: Stochastic Gradient Descent with Warm Restarts

Context Encoders: Feature Learning by Inpainting

Deep Networks with Stochastic Depth

Packet Loss Concealment Based on Deep Neural Networks for Digital Speech Transmission

Deep Residual Learning for Image Recognition

ESC: Dataset for Environmental Sound Classification

Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion

Extracting and composing robust features with denoising autoencoders

Histograms of oriented gradients for human detection

Equal-loudness-level contours for pure tones.

Speech recognition

Signal estimation from modified short-time Fourier transform

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Conference on Neural Information Processing Systems

Proceedings, Part I, ser

A novel method of artificial bandwidth extension using deep architecture

Dropout: a simple way to prevent neural networks from overfitting

The Kaldi Speech Recognition Toolkit

Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content

Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation

Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Checklist 1. For all authors

Did you discuss any potential negative societal impacts of your work

code, data, models) or curating/releasing new assets... (a) If your work uses existing assets

Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

If you used crowdsourcing or conducted research with human subjects

Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Ans: please check §4.3

c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Did you include the full text of instructions given to participants and screenshots

Field of Study

Computer ScienceEngineering

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

Neural Information Processing Systems

Type

conference

URL

http://neurips.cc/

Alternate Names

Neural Inf Process Syst
NeurIPS
NIPS