Neural Voice Cloning with a Few Samples

Published in

Neural Information Processing Systems(2018)

External Links:

Generate Graph

TL;DR

While speaker adaptation can achieve better naturalness and similarity, the cloning time or required memory for the speaker encoding approach is significantly less, making it favorable for low-resource deployment.

Abstract

Voice cloning is a highly desired feature for personalized speech interfaces. Neural network based speech synthesis has been shown to generate high quality speech for a large number of speakers. In this paper, we introduce a neural voice cloning system that takes a few audio samples as input. We study two approaches: speaker adaptation and speaker encoding. Speaker adaptation is based on fine-tuning a multi-speaker generative model with a few cloning samples. Speaker encoding is based on training a separate model to directly infer a new speaker embedding from cloning audios and to be used with a multi-speaker generative model. In terms of naturalness of the speech and its similarity to original speaker, both approaches can achieve good performance, even with very few cloning audios. While speaker adaptation can achieve better naturalness and similarity, the cloning time or required memory for the speaker encoding approach is significantly less, making it favorable for low-resource deployment.

Authors

Jitong Chen

3 papers

Yanqi Zhou

3 papers

Sercan Ö. Arik

9 papers

References50 items

CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92)

Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions

Multi-content GAN for Few-Shot Font Style Transfer

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning

Neural Voice Cloning with a Few Samples

Published in

Neural Information Processing Systems(2018)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Jitong Chen

3 papers

Yanqi Zhou

3 papers

Sercan Ö. Arik

9 papers

References50 items

CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92)

Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions

Multi-content GAN for Few-Shot Font Style Transfer

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning

Kainan Peng

1 papers

Deep Voice 3: 2000-Speaker Neural Text-to-Speech

Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions

Embedding-Based Speaker Adaptive Training of Deep Neural Networks

Regularizing and Optimizing LSTM Language Models

VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop

Voice Synthesis for in-the-Wild Speakers via a Phonological Loop

Attention is All you Need

Deep Voice 2: Multi-Speaker Neural Text-to-Speech

Deep Speaker: an End-to-End Neural Speaker Embedding System

Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders

Tacotron: A Fully End-to-End Text-To-Speech Synthesis Model

Deep Voice: Real-time Neural Text-to-Speech

Char2Wav: End-to-End Speech Synthesis

Deep neural network-based speaker embeddings for end-to-end speaker verification

SampleRNN: An Unconditional End-to-End Neural Audio Generation Model

SUPERSEDED - CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit

Analysis of the Voice Conversion Challenge 2016 Evaluation Results

WaveNet: A Generative Model for Raw Audio

Locally Linear Embedding for Exemplar-Based Spectral Conversion

Conditional Image Generation with PixelCNN Decoders

Voice Morphing that improves TTS quality using an optimal dynamic frequency warping-and-weighting transform

Speaker adaptive training in deep neural networks using speaker dependent bottleneck features

One-Shot Generalization in Deep Generative Models

Exploring the Limits of Language Modeling

Human-level concept learning through probabilistic program induction

Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin

A probabilistic interpretation for artificial neural network-based voice conversion

Speaker Adaptive Training of Deep Neural Network Acoustic Models Using I-Vectors

Speaker diarization through speaker embeddings

A study of speaker adaptation for DNN-based speech synthesis

Librispeech: An ASR corpus based on public domain audio books

Voice Conversion Using Deep Neural Networks With Layer-Wise Generative Training

Fast Adaptation of Deep Neural Network Based on Discriminant Codes for Speech Recognition

One-shot learning by inverting a compositional causal process

Fast speaker adaptation of hybrid NN/HMM model for speech recognition based on discriminative learning of speaker code

KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition

Spectral Mapping Using Artificial Neural Networks for Voice Conversion

Probabilistic Linear Discriminant Analysis for Inferences About Identity

Statistical Parametric Speech Synthesis

ATR μ-talk speech synthesis system

Let us/encourage each other%. Figure 7: The sentences used to generate test samples for the voice cloning models. The white space characters / and % follow the same deﬁnition as in Ping et al.

On speaker adaptation of long short-term memory recurrent neural networks

Modeling speaker variability using long short-term memory networks for speech recognition

One-shot learning of generative speech concepts

Analysis of Speaker Adaptation Algorithms for HMM-Based Speech Synthesis and a Constrained SMAPLR Adaptation Algorithm

Field of Study

Computer ScienceEngineering

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

Neural Information Processing Systems

Type

conference

URL

http://neurips.cc/

Alternate Names

Neural Inf Process Syst
NeurIPS
NIPS