VampNet: Music Generation via Masked Acoustic Token Modeling (2023-07-10T00:00:00.000000Z)

TL;DR

It is shown that by prompting VampNet in various ways, it can apply it to tasks like music compression, inpainting, out Painting, continuation, and looping with variation (vamping), and is capable of maintaining style, genre, instrumentation, and other high-level aspects of the music.

Abstract

We introduce VampNet, a masked acoustic token modeling approach to music synthesis, compression, inpainting, and variation. We use a variable masking schedule during training which allows us to sample coherent music from the model by applying a variety of masking approaches (called prompts) during inference. VampNet is non-autoregressive, leveraging a bidirectional transformer architecture that attends to all tokens in a forward pass. With just 36 sampling passes, VampNet can generate coherent high-fidelity musical waveforms. We show that by prompting VampNet in various ways, we can apply it to tasks like music compression, inpainting, outpainting, continuation, and looping with variation (vamping). Appropriately prompted, VampNet is capable of maintaining style, genre, instrumentation, and other high-level aspects of the music. This flexible prompting capability makes VampNet a powerful music co-creation tool. Code and audio samples are available online.

Authors

Prem Seetharaman

5 papers

Bryan Pardo

3 papers

Rithesh Kumar

5 papers

TL;DR

Abstract

Authors

References35 items

High-Fidelity Audio Compression with Improved RVQGAN

Simple and Controllable Music Generation

SoundStorm: Efficient Parallel Audio Generation

MusicLM: Generating Music From Text

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

Muse: Text-To-Image Generation via Masked Generative Transformers

High Fidelity Neural Audio Compression

AudioLM: A Language Modeling Approach to Audio Generation

Efficient Training of Language Models to Fill in the Middle

MaskGIT: Masked Generative Image Transformer

High-Resolution Image Synthesis with Latent Diffusion Models

Masked Autoencoders Are Scalable Vision Learners

WaveBeat: End-to-end beat and downbeat tracking in the time domain

w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training

SoundStream: An End-to-End Neural Audio Codec

Structured Denoising Diffusion Models in Discrete State-Spaces

Taming Transformers for High-Resolution Image Synthesis

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Denoising Diffusion Probabilistic Models

Jukebox: A Generative Model for Music

Generative Modeling by Estimating Gradients of the Data Distribution

Low Bit-rate Speech Coding with VQ-VAE and a WaveNet Decoder

Self-Attention with Relative Position Representations

Fixing Weight Decay Regularization in Adam

Neural Discrete Representation Learning

Attention is All you Need

VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning

Improved Techniques for Training GANs

Fast Text-Conditional Discrete Denoising on Vector-Quantized Latent Spaces

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Improving Language Understanding by Generative Pre-Training

Statistical Theory of Extreme Values and Some Practical Applications : A Series of Lectures

Preﬁx and sufﬁx (inpaint) prompts: a segment at the beginning and at the end of the sequence is left un-masked

Periodic prompt: every P th timestep is left un-masked

Proceedings of the 24th ISMIR Conference,

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names