Scaling Speech Technology to 1, 000+ Languages (2023-05-22T00:00:00.000000Z)

TL;DR

Experiments show that the multilingual speech recognition model more than halves the word error rate of Whisper on 54 languages of the FLEURS benchmark while being trained on a small fraction of the labeled data.

Abstract

Expanding the language coverage of speech technology has the potential to improve access to information for many more people. However, current speech technology is restricted to about one hundred languages which is a small fraction of the over 7,000 languages spoken around the world. The Massively Multilingual Speech (MMS) project increases the number of supported languages by 10-40x, depending on the task. The main ingredients are a new dataset based on readings of publicly available religious texts and effectively leveraging self-supervised learning. We built pre-trained wav2vec 2.0 models covering 1,406 languages, a single multilingual automatic speech recognition model for 1,107 languages, speech synthesis models for the same number of languages, as well as a language identification model for 4,017 languages. Experiments show that our multilingual speech recognition model more than halves the word error rate of Whisper on 54 languages of the FLEURS benchmark while being trained on a small fraction of the labeled data.

Authors

Vineel Pratap

3 papers

Paden Tomasello

3 papers

Michael Auli

14 papers

TL;DR

Abstract

Authors

References107 items

Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling

Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages

Improving Massively Multilingual ASR with Auxiliary CTC Objectives

Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining

Robust Speech Recognition via Large-Scale Weak Supervision

Massively Multilingual ASR on 70 Languages: Tokenization, Architecture, and Generalization Capabilities

Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-to-Speech

Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Tasks

ASR2K: Speech Recognition for Around 2000 Languages without Audio

BibleTTS: a large, high-fidelity, multilingual, and uniquely African speech corpus

FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech

Building Machine Translation Systems for the Next Thousand Languages

MAESTRO: Matched Speech Text Representations through Modality Matching

Towards End-to-End Unsupervised Speech Recognition

mSLAM: Massively multilingual joint pre-training for speech and text

Flashlight: Enabling Innovation in Tools for Machine Learning

Global predictors of language endangerment and the future of linguistic diversity

YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone

XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale

Towards Building ASR Systems for the Next Billion Users

Pseudo-Labeling for Massively Multilingual Speech Recognition

Torchaudio: Building Blocks for Audio and Speech Processing

Improved Language Identification Through Cross-Lingual Self-Supervised Learning

A Survey on Neural Speech Synthesis

Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech

SpeechBrain: A General-Purpose Speech Toolkit

Unsupervised Speech Recognition

Scaling End-to-End Models for Large-Scale Multilingual ASR

ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep learning

SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network

Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training

Multilingual Byte2Speech Models for Scalable Low-resource Speech Synthesis

VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation

Exploring wav2vec 2.0 on speaker verification and language identification

VOXLINGUA107: A Dataset for Spoken Language Recognition

MLS: A Large-Scale Multilingual Dataset for Speech Research

Rethinking Evaluation in ASR: Are Our Models Robust Enough?

HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

Phonological Features for 0-shot Multilingual Speech Synthesis

One Model, Many Languages: Meta-learning for Multilingual Text-to-Speech

CTC-Segmentation of Large Corpora for German End-to-End Speech Recognition

Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters

Unsupervised Cross-lingual Representation Learning for Speech Recognition

Real Time Speech Enhancement in the Waveform Domain

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

The Johns Hopkins University Bible Corpus: 1600+ Tongues for Typological Exploration

Racial disparities in automated speech recognition

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Multilingual Denoising Pre-training for Neural Machine Translation

Common Voice: A Massively-Multilingual Speech Corpus

Unsupervised Cross-lingual Representation Learning at Scale

vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations

Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model

Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning

CMU Wilderness Multilingual Speech Dataset

Who Needs Words? Lexicon-Free Speech Recognition

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Parameter-Efficient Transfer Learning for NLP

Cross-lingual Language Model Pretraining

Wav2Letter++: A Fast Open-source Speech Recognition System

Bytes Are All You Need: End-to-end Multilingual Speech Recognition and Synthesis with Bytes

Multilingual Sequence-to-Sequence Speech Recognition: Architecture, Transfer Learning, and Language Modeling

Representation Learning with Contrastive Predictive Coding

Out-of-the-box Universal Romanization Tool uroman

The challenge of realistic music generation: modelling raw audio at scale

ESPnet: End-to-End Speech Processing Toolkit

Multilingual Speech Recognition with a Single End-to-End Model

Attention is All you Need

Categorical Reparameterization with Gumbel-Softmax

Wav2Letter: an End-to-End ConvNet-based Speech Recognition System

Layer Normalization

Density estimation using Real NVP

Training Deep Nets with Sublinear Memory Cost

Adam: A Method for Stochastic Optimization

A massively parallel corpus: the Bible in 100 languages

Sequence to Sequence Learning with Neural Networks