Msanii: High Fidelity Music Synthesis on a Shoestring Budget (2023-01-16T00:00:00.000000Z)

TL;DR

This work presents Msanii, a novel diffusion-based model for synthesizing long-context, high-fidelity music efficiently that combines the expressiveness of mel spectrograms, the generative capabilities of diffusion models, and the vocoding capabilities of neural vocoders.

Abstract

In this paper, we present Msanii, a novel diffusion-based model for synthesizing long-context, high-fidelity music efficiently. Our model combines the expressiveness of mel spectrograms, the generative capabilities of diffusion models, and the vocoding capabilities of neural vocoders. We demonstrate the effectiveness of Msanii by synthesizing tens of seconds (190 seconds) of stereo music at high sample rates (44.1 kHz) without the use of concatenative synthesis, cascading architectures, or compression techniques. To the best of our knowledge, this is the first work to successfully employ a diffusion-based model for synthesizing such long music samples at high sample rates. Our demo can be found https://kinyugo.github.io/msanii-demo and our code https://github.com/Kinyugo/msanii .

Authors

Kinyugo Maina

1 papers

TL;DR

Abstract

Authors

References44 items

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

Mel Spectrogram Inversion with Stable Pitch

Understanding Diffusion Models: A Unified Perspective

Musika! Fast Infinite Waveform Music Generation

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Hierarchical Text-Conditional Image Generation with CLIP Latents

ISTFTNET: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform

RePaint: Inpainting using Denoising Diffusion Probabilistic Models

A ConvNet for the 2020s

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

High-Resolution Image Synthesis with Latent Diffusion Models

Generative Adversarial Networks

Variational Diffusion Models

CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis

Diffusion Models Beat GANs on Image Synthesis

DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism

Improved Denoising Diffusion Probabilistic Models

Denoising Diffusion Implicit Models

DiffWave: A Versatile Diffusion Model for Audio Synthesis

POP909: A Pop-Song Dataset for Music Arrangement Generation

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Denoising Diffusion Probabilistic Models

Language Models are Few-Shot Learners

Jukebox: A Generative Model for Music

Mel-spectrogram augmentation for sequence to sequence voice conversion

Analyzing and Improving the Image Quality of StyleGAN

MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

MelNet: A Generative Model for Audio in the Frequency Domain

Adversarial Generation of Time-Frequency Features with application in audio synthesis

GANSynth: Adversarial Neural Audio Synthesis

Fast Spectrogram Inversion Using Multi-Head Convolutional Neural Networks

The challenge of realistic music generation: modelling raw audio at scale

Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions

Attention is All you Need

Instance Normalization: The Missing Ingredient for Fast Stylization

Variational Inference with Normalizing Flows

U-Net: Convolutional Networks for Biomedical Image Segmentation

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

A fast Griffin-Lim algorithm

Signal estimation from modified short-time Fourier transform

Diffusers: State-of-the-art diffusion models. https: //github.com/huggingface/diffusers

What are diffusion models? lilian-weng

AUTO-ENCODING VARIATIONAL BAYES

Fluidsynth

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names