DC CoMix TTS: An End-to-End Expressive TTS with Discrete Code Collaborated with Mixer (2023-05-31T00:00:00.000000Z)

TL;DR

A new input representation and simple architecture are proposed to achieve improved prosody modeling in TTS and it is demonstrated that the reference encoder learns better speaker-independent prosody when discrete code is utilized as input in the experiments.

Abstract

Despite the huge successes made in neutral TTS, content-leakage remains a challenge. In this paper, we propose a new input representation and simple architecture to achieve improved prosody modeling. Inspired by the recent success in the use of discrete code in TTS, we introduce discrete code to the input of the reference encoder. Specifically, we leverage the vector quantizer from the audio compression model to exploit the diverse acoustic information it has already been trained on. In addition, we apply the modified MLP-Mixer to the reference encoder, making the architecture lighter. As a result, we train the prosody transfer TTS in an end-to-end manner. We prove the effectiveness of our method through both subjective and objective evaluations. We demonstrate that the reference encoder learns better speaker-independent prosody when discrete code is utilized as input in the experiments. In addition, we obtain comparable results even when fewer parameters are inputted.

Authors

Yerin Choi

1 papers

M. Koo

1 papers

TL;DR

Abstract

Authors

References32 items

A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

High Fidelity Neural Audio Compression

AudioLM: A Language Modeling Approach to Audio Generation

DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders

Prosodyspeech: Towards Advanced Prosody Model for Neural Text-to-Speech

FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis

VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature

JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech

BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis

Fine-Grained Style Control In Transformer-Based Text-To-Speech Synthesis

Mixer-TTS: Non-Autoregressive, Fast and Compact Text-to-Speech Model Conditioned on Language Model Embeddings

PortaSpeech: Portable and High-Quality Generative Text-to-Speech

SoundStream: An End-to-End Neural Audio Codec

Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech

MLP-Mixer: An all-MLP Architecture for Vision

HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Denoising Diffusion Probabilistic Models

FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

End-to-End Adversarial Text-to-Speech

Unsupervised Style and Content Separation by Minimizing Mutual Information for Speech Synthesis

Espnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit

NeMo: a toolkit for building AI applications using Neural Modules

Spoken Language Recognition using X-vectors

Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis

Decoupled Weight Decay Regularization

Tacotron: Towards End-to-End Speech Synthesis

SUPERSEDED - CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit

Variational Inference with Normalizing Flows

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

RAD-TTS: Parallel Flow-Based TTS with Robust Alignment Learning and Diverse Synthesis

Field of Study

Venue Information

Name

Type

URL

Alternate Names