ALCAP: Alignment-Augmented Music Captioner (2022-12-21T00:00:00.000000Z)

TL;DR

This study introduces a method to systematically learn multimodal alignment between audio and lyrics through contrastive learning, which paves the way for models to achieve deeper cross-modal coherence, thereby producing high-quality captions.

Abstract

Music captioning has gained significant attention in the wake of the rising prominence of streaming media platforms. Traditional approaches often prioritize either the audio or lyrics aspect of the music, inadvertently ignoring the intricate interplay between the two. However, a comprehensive understanding of music necessitates the integration of both these elements. In this study, we delve into this overlooked realm by introducing a method to systematically learn multimodal alignment between audio and lyrics through contrastive learning. This not only recognizes and emphasizes the synergy between audio and lyrics but also paves the way for models to achieve deeper cross-modal coherence, thereby producing high-quality captions. We provide both theoretical and empirical results demonstrating the advantage of the proposed method, which achieves new state-of-the-art on two music captioning datasets.

Authors

Changyou Chen

6 papers

Zihao He

2 papers

Kristina Lerman

2 papers

TL;DR

Abstract

Authors

References39 items

X-Norm: Exchanging Normalization Parameters for Bimodal Fusion

Interpreting Song Lyrics with an Audio-Informed Pre-trained Language Model

ReLyMe: Improving Lyric-to-Melody Generation by Incorporating Lyric-Melody Relationships

Large-Scale Retrieval for Reinforcement Learning

CoCa: Contrastive Captioners are Image-Text Foundation Models

A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing

Unsupervised Vision-and-Language Pretraining via Retrieval-based Multi-Granular Alignment

Retrieval-Augmented Reinforcement Learning

Cold Start Similar Artists Ranking with Gravity-Inspired Graph Autoencoders

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation

MusCaps: Generating Captions for Music Audio

Learning Transferable Visual Models From Natural Language Supervision

Music4All: A New Music Database and Its Applications

CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information

PyTorch: An Imperative Style, High-Performance Deep Learning Library

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

HuggingFace's Transformers: State-of-the-art Natural Language Processing

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Toward Interpretable Music Tagging with Self-Attention

Moodplay: Interactive music recommendation based on Artists' mood similarity

Improving Reinforcement Learning Based Image Captioning with Natural Language Prior

Representation Learning with Contrastive Predictive Coding

Mutual Information Neural Estimation

Deep Reinforcement Learning-Based Image Captioning with Embedding Reward

Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

Probabilistic Graphical Models: Principles and Techniques - Adaptive Computation and Machine Learning

METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments

ROUGE: A Package for Automatic Evaluation of Summaries

The information bottleneck method

Long Short-Term Memory

Deep Variational Information Bottleneck

The Million Song Dataset

Relationships Between Lyrics and Melody in Popular Music

Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization

Since we know that both I(x, z) and I(m; t) must be non-negative, we have I(x; z) = I(m; t) = 0

− I(m; t)

This completes the proof of Proposition 4.1. Based on the above derivcation, if (m, t) pairs are sampled randomly

this leads to the independency of x and z , i.e, , z contains zero information of z . This completes the proof of Proposition 4.2

=E p(x,z) log p(x|z) p(x)

Field of Study

Venue Information

Name

Type

URL

Alternate Names