Audioclip: Extending Clip to Image, Text and Audio (2021-06-24T00:00:00.000000Z)

TL;DR

An extension of the CLIP model that handles audio in addition to text and images that achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task and out-performs others by reaching accuracies of 97.15 % on ESC-50 and 90.07 % on UrbanSound8K.

Abstract

The rapidly evolving field of sound classification has greatly benefited from the methods of other domains. Today, the trend is to fuse domain-specific tasks and approaches together, which provides the community with new outstanding models.We present AudioCLIP – an extension of the CLIP model that handles audio in addition to text and images. Utilizing the AudioSet dataset, our proposed model incorporates the ESResNeXt audio-model into the CLIP framework, thus enabling it to perform multimodal classification and keeping CLIP’s zero-shot capabilities.AudioCLIP achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task and out-performs others by reaching accuracies of 97.15 % on ESC-50 and 90.07 % on UrbanSound8K. Further, it sets new baselines in the zero-shot ESC-task on the same datasets (69.40 % and 68.78 %, respectively).We also asses the influence of different training setups on the final performance of the proposed model. For the sake of reproducibility, our code is published.

Authors

A. Dengel

8 papers

A. Guzhov

1 papers

Federico Raue

1 papers

TL;DR

Abstract

Authors

References34 items

ERANNs: Efficient Residual Audio Neural Networks for Audio Pattern Recognition

Enhancing Audio-Visual Association with Self-Supervised Curriculum Learning

Multimodal Self-Supervised Learning of General Audio Representations

ESResNe(X)t-fbsp: Learning Robust Time-Frequency Transformation of Audio

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

AST: Audio Spectrogram Transformer

MDMMT: Multidomain Multimodal Transformer for Video Retrieval

Learning Transferable Visual Models From Natural Language Supervision

Zero-Shot Audio Classification Via Semantic Embeddings

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Rethinking CNN Models for Audio Classification

A Sequential Self Teaching Approach for Improving Generalization in Sound Event Recognition

Self-Supervised MultiModal Versatile Networks

ESResNet: Environmental Sound Classification Based on Visual Domain Models

AudioCaps: Generating Captions for Audios in The Wild

Zero-Shot Audio Classification Based On Class Label Embeddings

SoundSemantics: Exploiting Semantic Knowledge in Text for Embedded Acoustic Event Classification

Learning from Between-class Examples for Deep Sound Recognition

Unsupervised Filterbank Learning Using Convolutional Restricted Boltzmann Machine for Environmental Sound Classification

Attention is All you Need

Audio Set: An ontology and human-labeled dataset for audio events

Learning environmental sounds with end-to-end convolutional neural network

Xception: Deep Learning with Depthwise Separable Convolutions

Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification

Deep Residual Learning for Image Recognition

Environmental sound classification with convolutional neural networks

ESC: Dataset for Environmental Sound Classification

A Dataset and Taxonomy for Urban Sound Research

ImageNet: A large-scale hierarchical image database

Acceleration of stochastic approximation by averaging

Language Models are Unsupervised Multitask Learners

Differential Data Augmentation Techniques for Medical Imaging Classification Tasks

Computational signal processing with wavelets, vol

A method of solving a convex programming problem with convergence rate o (1/kˆ 2) o (1/k2)

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names