Temporal and cross-modal attention for audio-visual zero-shot learning (2022-07-20T00:00:00.000000Z)

TL;DR

It is shown that the proposed framework that ingests temporal features yields state-of-the-art performance on the \ucf, \vgg, and \activity benchmarks for (generalised) zero-shot learning.

Abstract

Audio-visual generalised zero-shot learning for video classification requires understanding the relations between the audio and visual information in order to be able to recognise samples from novel, previously unseen classes at test time. The natural semantic and temporal alignment between audio and visual data in video data can be exploited to learn powerful representations that generalise to unseen classes at test time. We propose a multi-modal and Temporal Cross-attention Framework (\modelName) for audio-visual generalised zero-shot learning. Its inputs are temporally aligned audio and visual features that are obtained from pre-trained networks. Encouraging the framework to focus on cross-modal correspondence across time instead of self-attention within the modalities boosts the performance significantly. We show that our proposed framework that ingests temporal features yields state-of-the-art performance on the \ucf, \vgg, and \activity benchmarks for (generalised) zero-shot learning. Code for reproducing all results is available at \url{https://github.com/ExplainableML/TCAF-GZSL}.

Authors

Zeynep Akata

19 papers

A. S. Koepke

8 papers

Thomas Hummel

2 papers

TL;DR

Abstract

Authors

References78 items

Crossmodal Representation Learning for Zero-shot Action Recognition

Audiovisual Generalised Zero-shot Learning with Cross-modal Attention and Language

Distilling Audio-Visual Knowledge by Compositional Contrastive Learning

T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval

Self-supervised object detection from audio-visual correspondence

Strumming to the Beat: Audio-Conditioned Contrastive Video Textures

Localizing Visual Sounds the Hard Way

HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval

Perceiver: General Perception with Iterative Attention

Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Attribute Prototype Network for Zero-Shot Learning

Self-Supervised Learning of Audio-Visual Objects from Video

Multi-modal Transformer for Video Retrieval

Foley Music: Learning to Generate Music from Videos

Labelling unlabelled videos from scratch with multi-modal self-supervision

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data

AVGZSLNet: Audio-Visual Generalized Zero-Shot Learning by Reconstructing Label Features from Multi-Modal Embeddings

A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer

Sight to Sound: An End-to-End Approach for Visual Piano Transcription

Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification

Multi-modal Self-Supervision from Generalized Data Transformations

Rethinking Zero-Shot Video Classification: End-to-End Training for Realistic Applications

ASR is All You Need: Cross-Modal Distillation for Lip Reading

Self-Supervised Learning by Cross-Modal Audio-Video Clustering

Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zero-shot Classification and Retrieval of Videos

Audiovisual Transformer Architectures for Large-Scale Classification and Synchronization of Weakly Labeled Audio Events

Vision-Infused Deep Audio Inpainting

Attribute Attention for Semantic Disambiguation in Zero-Shot Learning

Learning Video Representations using Contrastive Bidirectional Transformer

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

VisualBERT: A Simple and Performant Baseline for Vision and Language

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Attentive Region Embedding Network for Zero-Shot Learning

Learning Feature-to-Feature Translator by Alternating Back-Propagation for Generative Zero-Shot Learning

Co-Separating Sounds of Visual Objects

VideoBERT: A Joint Model for Video and Language Representation Learning

F-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning

You Said That?: Synthesising Talking Faces from Audio

Deep Audio-Visual Speech Recognition

X2Face: A network for controlling face generation by using images, audio, and pose codes

Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization

Audio-Visual Scene Analysis with Self-Supervised Multisensory Features

Audio-Visual Event Localization in Unconstrained Videos

Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning

Objects that Sound

Feature Generating Networks for Zero-Shot Learning

Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly

Attention is All you Need

SoundNet: Learning Sound Representations from Unlabeled Video

CNN architectures for large-scale audio classification

Layer Normalization

Gaussian Error Linear Units (GELUs)

An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild

Label-Embedding for Image Classification

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Adam: A Method for Stochastic Optimization

Learning Spatiotemporal Features with 3D Convolutional Networks

Evaluation of output embeddings for fine-grained image classification

Large-Scale Video Classification with Convolutional Neural Networks

DeViSE: A Deep Visual-Semantic Embedding Model

Efficient Estimation of Word Representations in Vector Space

The Caltech-UCSD Birds-200-2011 Dataset

Rectified Linear Units Improve Restricted Boltzmann Machines

SUN database: Large-scale scene recognition from abbey to zoo

Describing objects by their attributes

Reformulating Zero-shot Action Recognition for Multi-label Actions

Audiovisual Transformer with Instance Attention for Audio-Visual Event Localization

Visual Pitch Estimation

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Guitar Music Transcription from Silent Video

Dropout: a simple way to prevent neural networks from overfitting