ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit (2023-04-10T00:00:00.000000Z)

TL;DR

The overall design, example models for each task, and performance benchmarking behind ESPnet-ST-v2 are described, which is publicly available at https://github.com/espnet/esp net.

Abstract

ESPnet-ST-v2 is a revamp of the open-source ESPnet-ST toolkit necessitated by the broadening interests of the spoken language translation community. ESPnet-ST-v2 supports 1) offline speech-to-text translation (ST), 2) simultaneous speech-to-text translation (SST), and 3) offline speech-to-speech translation (S2ST) – each task is supported with a wide variety of approaches, differentiating ESPnet-ST-v2 from other open source spoken language translation toolkits. This toolkit offers state-of-the-art architectures such as transducers, hybrid CTC/attention, multi-decoders with searchable intermediates, time-synchronous blockwise CTC/attention, Translatotron models, and direct discrete unit models. In this paper, we describe the overall design, example models for each task, and performance benchmarking behind ESPnet-ST-v2, which is publicly available at https://github.com/espnet/espnet.

Authors

Shinji Watanabe

12 papers

J. Pino

8 papers

Siddharth Dalmia

3 papers

TL;DR

Abstract

Authors

References64 items

UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units

Speech-to-Speech Translation For A Real-world Unwritten Language

CTC Alignments Improve Autoregressive Translation

E-Branchformer: Branchformer with Enhanced Merging for Speech Recognition

Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding

Quality-Aware Decoding for Neural Machine Translation

Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation

CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022

Large-Scale Streaming End-to-End Speech Translation with Neural Transducers

Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translation

CVSS Corpus and Massively Multilingual Speech-to-Speech Translation

Textless Speech-to-Speech Translation on Real Data

Torchaudio: Building Blocks for Audio and Speech Processing

ESPnet2-TTS: Extending the Edge of TTS Research

Translatotron 2: High-quality direct speech-to-speech translation with voice preservation

Direct Speech-to-Speech Translation With Discrete Units

Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR

Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?

Streaming cascade-based speech translation leveraged by a direct segmentation model

SUPERB: Speech processing Universal PERformance Benchmark

Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks

Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation

Speech Resynthesis from Discrete Disentangled Self-Supervised Representations

Intermediate Loss Regularization for CTC-Based Speech Recognition

CTC-based Compression for Direct Speech Translation

NeurST: Neural Speech Translation Toolkit

SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation

Recent Developments on Espnet Toolkit Boosted By Conformer

HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq

Transformers: State-of-the-Art Natural Language Processing

SIMULEVAL: An Evaluation Toolkit for Simultaneous Translation

Streaming Transformer Asr With Blockwise Synchronous Beam Search

UWSpeech: Speech to Speech Translation for Unwritten Languages

Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection

Conformer: Convolution-augmented Transformer for Speech Recognition

Alignment-Length Synchronous Decoding for RNN Transducer

ESPnet-ST: All-in-One Speech Translation Toolkit

Multilingual Denoising Pre-training for Neural Machine Translation

Espnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit

NeMo: a toolkit for building AI applications using Neural Modules

A Comparative Study on Transformer vs RNN in Speech Applications

MuST-C: a Multilingual Speech Translation Corpus

Direct speech-to-speech translation with a sequence-to-sequence model

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling

Open Source Toolkit for Speech to Text Translation

Hierarchical Multitask Learning With CTC

A Call for Clarity in Reporting BLEU Scores

Hybrid CTC/Attention Architecture for End-to-End Speech Recognition

Attention is All you Need

Tacotron: Towards End-to-End Speech Synthesis

Bridges: a uniquely flexible HPC resource for new communities and data analytics

XSEDE: Accelerating Scientific Discovery

Statistical Machine Translation

Sequence Transduction with Recurrent Neural Networks

TONGUES: rapid development of a speech-to-speech translation system

JANUS: a speech-to-speech translation system using connectionist and symbolic processing strategies

CMU’s IWSLT 2022 Dialect Speech Translation System

The USTC-NELSLIP Offline Speech Translation Systems for IWSLT 2022

Findings of the IWSLT 2022 Evaluation Campaign

ESPnet-ST IWSLT 2021 Offline Speech Translation System

The IWSLT 2018 Evaluation Campaign

A historical perspective of speech recognition

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names