Dict-TTS: Learning to Pronounce with Prior Dictionary Knowledge for Text-to-Speech (2022-06-05T00:00:00.000000Z)

TL;DR

Dict-TTS is proposed, a semantic-aware generative text-to-speech model with an online website dictionary that outperforms several strong baseline models in terms of pronunciation accuracy and improves the prosody modeling of TTS systems.

Abstract

Polyphone disambiguation aims to capture accurate pronunciation knowledge from natural text sequences for reliable Text-to-speech (TTS) systems. However, previous approaches require substantial annotated training data and additional efforts from language experts, making it difficult to extend high-quality neural TTS systems to out-of-domain daily conversations and countless languages worldwide. This paper tackles the polyphone disambiguation problem from a concise and novel perspective: we propose Dict-TTS, a semantic-aware generative text-to-speech model with an online website dictionary (the existing prior information in the natural language). Specifically, we design a semantics-to-pronunciation attention (S2PA) module to match the semantic patterns between the input text sequence and the prior semantics in the dictionary and obtain the corresponding pronunciations; The S2PA module can be easily trained with the end-to-end TTS model without any annotated phoneme labels. Experimental results in three languages show that our model outperforms several strong baseline models in terms of pronunciation accuracy and improves the prosody modeling of TTS systems. Further extensive analyses demonstrate that each design in Dict-TTS is effective. The code is available at \url{https://github.com/Zain-Jiang/Dict-TTS}.

Authors

Yi Ren

10 papers

Jinglin Liu

9 papers

Ziyue Jiang

2 papers

TL;DR

Abstract

Authors

References79 items

ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech

GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis

SyntaSpeech: Syntax-Aware Generative Adversarial Text-to-Speech

FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis

Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speech

g2pW: A Conditional Weighted Softmax BERT for Polyphone Disambiguation in Mandarin

Polyphone Disambiguation and Accent Prediction Using Pre-Trained Language Models in Japanese TTS Front-End

Neural Lexicon Reader: Reduce Pronunciation Errors in End-to-end TTS by Leveraging External Textual Knowledge

WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition

PortaSpeech: Portable and High-Quality Generative Text-to-Speech

A Survey on Neural Speech Synthesis

Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech

Joint Alignment Learning-Attention Based Model for Grapheme-to-Phoneme Conversion

Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter

DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism

PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS

Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling

Using Prior Knowledge to Guide BERT’s Attention in Semantic Textual Matching Tasks

Unified Mandarin TTS Front-end Based on Distilled BERT Model

A Mask-Based Model for Mandarin Chinese Polyphone Disambiguation

HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis

FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

End-to-End Adversarial Text-to-Speech

Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search

Multilingual Grapheme-To-Phoneme Conversion with Byte Representation

g2pM: A Neural Grapheme-to-Phoneme Conversion Package for MandarinChinese Based on a New Open Benchmark Dataset

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Common Voice: A Massively-Multilingual Speech Corpus

Knowledge Distillation from Bert in Pre-Training and Fine-Tuning for Polyphone Disambiguation

Unsupervised Cross-lingual Representation Learning at Scale

Generalization through Memorization: Nearest Neighbor Language Models

Neural Machine Translation for Multilingual Grapheme-to-Phoneme Conversion

Disambiguation of Chinese Polyphones in an End-to-End Framework with Semantic Features Extracted by Pre-Trained BERT

Pre-Trained Text Embeddings for Enhanced Text-to-Speech Synthesis

Pre-Trained Text Representations for Improving Front-End Text Processing in Mandarin Text-to-Speech Synthesis

Polyphone Disambiguation for Mandarin Chinese Using Conditional Neural Network with Multi-level Embedding Features

What Does BERT Look at? An Analysis of BERT’s Attention

A Structural Probe for Finding Syntax in Word Representations

BERT Rediscovers the Classical NLP Pipeline

Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion

Learning pronunciation from a foreign language in speech synthesis networks

Neural Speech Synthesis with Transformer Network

Dissecting Contextual Word Embeddings: Architecture and Representation

Convolutional Sequence to Sequence Model with Non-Sequential Greedy Decoding for Grapheme to Phoneme Conversion

Self-Attention with Relative Position Representations

Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions

JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesis

Attention is All you Need

Tacotron: Towards End-to-End Speech Synthesis

Deep Voice: Real-time Neural Text-to-Speech

Categorical Reparameterization with Gumbel-Softmax

A bi-directional LSTM approach for polyphone disambiguation in Mandarin Chinese

WaveNet: A Generative Model for Raw Audio

Sequence-to-sequence neural net models for grapheme-to-phoneme conversion

Polyphonic Word Disambiguation with Machine Learning Approaches

Disambiguating effectively Chinese polyphonic ambiguity based on unify approach

Information retrieval for music and motion

Principles of generative phonology : an introduction

Issues in building general letter to sound rules

Acknowledgments

Heteronyms and polyphones: Categories of words with multiple phonemic representations

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Chinese standard mandarin speech corpus

INTERSPEECH 2014 15th Annual Conference of the International Speech Communication Association

Disambiguation of chinese polyphonic characters

Dynamic Time Warping

Figure 7: Screenshots of audio quality and prosody evaluations

Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]

Did you state the full set of assumptions of all theoretical results

data, models) or curating/releasing new assets... (a) If your work uses existing assets

Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix H in the supplementary materials

you used crowdsourcing or conducted research with human subjects

b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Have you read the ethics review guidelines and ensured that your paper conforms to them