Learning to Separate Object Sounds by Watching Unlabeled Video (2018-04-05T00:00:00.000000Z)

TL;DR

This work is the first to learn audio source separation from large-scale "in the wild" videos containing multiple audio sources per video, and obtains state-of-the-art results on visually-aided audio sources separation and audio denoising.

Abstract

Perceiving a scene most fully requires all the senses. Yet modeling how objects look and sound is challenging: most natural scenes and events contain multiple objects, and the audio track mixes all the sound sources together. We propose to learn audio-visual object models from unlabeled video, then exploit the visual context to perform audio source separation in novel videos. Our approach relies on a deep multi-instance multi-label learning framework to disentangle the audio frequency bases that map to individual visual objects, even without observing/hearing those objects in isolation. We show how the recovered disentangled bases can be used to guide audio source separation to obtain better-separated, object-level sounds. Our work is the first to learn audio source separation from large-scale "in the wild" videos containing multiple audio sources per video. We obtain state-of-the-art results on visually-aided audio source separation and audio denoising. Our video results: this http URL

Authors

K. Grauman

10 papers

Ruohan Gao

5 papers

R. Feris

10 papers

TL;DR

Abstract

Authors

References89 items

The Conversation: Deep Audio-Visual Speech Enhancement

Audio-Visual Scene Analysis with Self-Supervised Multisensory Features

Looking to listen at the cocktail party

The Sound of Pixels

Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input

Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning

Objects that Sound

Visual to Sound: Generating Natural Sound for Videos in the Wild

Visual Speech Enhancement

Visual Speech Enhancement using Noise-Invariant Training

Generative Modeling of Audible Shapes for Object Perception

Truly Multi-modal YouTube-8M Video Classification with Video, Audio, and Text

See, Hear, and Read: Deep Aligned Representations

Look, Listen and Learn

Deep Cross-Modal Audio-Visual Generation

UntrimmedNets for Weakly Supervised Action Recognition and Detection

Audio-visual object localization and separation using low-rank and sparsity

Audio Set: An ontology and human-labeled dataset for audio events

See and listen: Score-informed association of sound tracks to players in chamber music performance videos

MIML-FCN+: Multi-Instance Multi-Label Learning via Fully Convolutional Networks with Privileged Information

Deep MIML Network

Learning Word-Like Units from Joint Audio-Visual Analysis

SoundNet: Learning Sound Representations from Unlabeled Video

Two multimodal approaches for single microphone source separation

Visually Indicated Sounds

Deep Residual Learning for Image Recognition

Weakly Supervised Deep Detection Networks

Deep clustering: Discriminative embeddings for segmentation and separation

Deep multiple instance learning for image classification and auto-annotation

Anticipating Visual Representations from Unlabeled Video

NMF-based blind source separation using a linear predictive coding error clustering criterion

Deep Karaoke: Extracting Vocals from Musical Mixtures Using a Convolutional Deep Neural Network

Weakly Supervised Object Localization with Multi-Fold Multiple Instance Learning

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Adam: A Method for Stochastic Optimization

Deep visual-semantic alignments for generating image descriptions

Long-term recurrent convolutional networks for visual recognition and description

What's Making that Sound?

Text-Informed Audio Source Separation. Example-Based Approach Using Non-Negative Matrix Partial Co-Factorization

Kernel Additive Models for Source Separation

Deep learning for monaural speech separation

An interactive audio source separation framework based on non-negative matrix factorization

Discovering joint audio–visual codewords for video event detection

Multimodal Analysis for Identification and Segmentation of Moving-Sounding Objects

NMF-based environmental sound source separation using time-variant gain features

Weakly Supervised Localization and Learning with Generic Knowledge

Score informed audio source separation using a parametric model of non-negative spectrogram

Clustering NMF basis functions using Shifted NMF for monaural sound source separation

JOINT AND INDIVIDUAL VARIATION EXPLAINED (JIVE) FOR INTEGRATED ANALYSIS OF MULTIPLE DATA TYPES.

Algorithms for Nonnegative Matrix Factorization with the β-Divergence

Blind Audiovisual Source Separation Based on Sparse Redundant Representations

Under-Determined Reverberant Audio Source Separation Using a Full-Rank Spatial Covariance Model

Nonnegative Matrix Factorization with the Itakura-Saito Divergence: With Application to Music Analysis

Keywords to visual categories: Multiple-instance learning forweakly supervised object categorization

Supervised and Semi-supervised Separation of Sounds from Single-Channel Mixtures

Harmony in Motion

Visual cues can modulate integration and segregation of objects in auditory scene analysis

Monaural Sound Source Separation by Nonnegative Matrix Factorization With Temporal Continuity and Sparseness Criteria

Evaluation campaigns and TRECVid

Performance measurement in blind audio source separation

Large-scale concept ontology for multimedia

Pixels that sound

Blind separation of speech mixtures via time-frequency masking

Names and faces in the news

Real-time speaker localization and speech separation by audio-visual integration

Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary

Ausio-visual Segmentation and "The Cocktail Party Effect"

Independent component analysis: algorithms and applications

Audio Vision: Using Audio-Visual Synchrony to Locate Sounds

Solving the Multiple Instance Problem with Axis-Parallel Rectangles

Signal estimation from modified short-time Fourier transform

Co-Training of Audio and Video Representations from Self-Co-Training of Audio and Video Representations from Self-Supervised Temporal Synchronization Supervised Temporal Synchronization

INFORMED AUDIO SOURCE SEPARATION

SOURCE-FILTER BASED CLUSTERING FOR MONAURAL BLIND SOURCE SEPARATION

Mixing Audiovisual Speech Processing and Blind Source Separation for the Extraction of Speech Signals From Convolutive Mixtures

INVESTIGATING SINGLE-CHANNEL AUDIO SOURCE SEPARATION METHODS BASED ON NON-NEGATIVE MATRIX FACTORIZATION