Solos: A Dataset for Audio-Visual Music Analysis (2020-06-14T00:00:00.000000Z)

TL;DR

This paper presents a new dataset of music performance videos, Solos, which can be used for training machine learning methods for multiple tasks such as audio-visual blind source separation and localization, cross-modal correspondences, cross themodal generation and, in general, any audio- visual self-supervised task.

Abstract

In this paper, we present a new dataset of music performance videos which can be used for training machine learning methods for multiple tasks such as audio-visual blind source separation and localization, cross-modal correspondences, cross-modal generation and, in general, any audio-visual self-supervised task. These videos, gathered from YouTube, consist of solo musical performances of 13 different instruments. Compared to previously proposed audio-visual datasets, Solos is cleaner since a big amount of its recordings are auditions and manually checked recordings, ensuring there is no background noise nor effects added in the video post-processing. Besides, it is, up to the best of our knowledge, the only dataset that contains the whole set of instruments present in the URMP [1] dataset, a high-quality dataset of 44 audio-visual recordings of multi-instrument classical music pieces with individual audio tracks. URMP was intented to be used for source separation, thus, we evaluate the performance on the URMP dataset of two different source-separation models trained on Solos. The dataset is publicly available at https://juanfmontesinos.github.io/Solos/

Authors

Juan F. Montesinos

4 papers

G. Haro

4 papers

Olga Slizovskaia

1 papers

TL;DR

Abstract

Authors

References43 items

Music Gesture for Visual Sound Separation

Vision-Infused Deep Audio Inpainting

Recursive Visual Sound Separation Using Minus-Plus Net

Interleaved Multitask Learning for Audio Source Separation with Independent Databases

Online Audio-Visual Source Association for Chamber Music Performances

Learning Individual Styles of Conversational Gesture

Speech2Face: Learning the Face Behind a Voice

Co-Separating Sounds of Visual Objects

The Sound of Motions

OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields

Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization

Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation

Audio-Visual Scene Analysis with Self-Supervised Multisensory Features

The Sound of Pixels

Photographic image synthesis with improved U-net

Audio to Body Dynamics

Objects that Sound

Visual to Sound: Generating Natural Sound for Videos in the Wild

Singing Voice Separation with Deep U-Net Convolutional Networks

Guiding audio source separation by video object information

Dilated Residual Networks

Deep Cross-Modal Audio-Visual Generation

See and listen: Score-informed association of sound tracks to players in chamber music performance videos

Monoaural Audio Source Separation Using Deep Convolutional Neural Networks

Creating a Multitrack Classical Music Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications

Image-to-Image Translation with Conditional Adversarial Networks

Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections

U-Net: Convolutional Networks for Biomedical Image Segmentation

Adam: A Method for Stochastic Optimization

Monaural Sound Source Separation by Nonnegative Matrix Factorization With Temporal Continuity and Sparseness Criteria

Performance measurement in blind audio source separation

Pixels that sound

Ausio-visual Segmentation and "The Cocktail Party Effect"

Independent component analysis: algorithms and applications

Audio Vision: Using Audio-Visual Synchrony to Locate Sounds

AUDIO-VISUAL SOURCE ASSOCIATION FOR STRING ENSEMBLES THROUGH MULTI-MODAL VIBRATO ANALYSIS

CHAPTER 7 – Frequency Domain Processing

on the Recognition of Speech, with

Mixing Audiovisual Speech Processing and Blind Source Separation for the Extraction of Speech Signals From Convolutive Mixtures

A Probabilistic Latent Variable Model for Acoustic Modeling

EURASIP Journal on Applied Signal Processing 2002:11, 1165–1173 c ○ 2002 Hindawi Publishing Corporation Separation of Audio-Visual Speech Sources: A New Approach Exploiting the Audio-Visual Coherence of Speech Stimuli

Independent Component Analysis: Blind source separation by sparse decomposition in a signal dictionary

PREDICTION-DRIVEN COMPUTATIONAL AUDITORY SCENE ANALYSIS FOR DENSE SOUND MIXTURES

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names