Learning Shared Semantic Space for Speech-to-Text Translation (2021-05-07T00:00:00.000000Z)

TL;DR

By projecting audio and text features to a common semantic representation, Chimera unifies MT and ST tasks and boosts the performance on ST benchmarks, MuST-C and Augmented Librispeech, to a new state-of-the-art.

Abstract

Having numerous potential applications and great impact, end-to-end speech translation (ST) has long been treated as an independent task, failing to fully draw strength from the rapid advances of its sibling - text machine translation (MT). With text and audio inputs represented differently, the modality gap has rendered MT data and its end-to-end models incompatible with their ST counterparts. In observation of this obstacle, we propose to bridge this representation gap with Chimera. By projecting audio and text features to a common semantic representation, Chimera unifies MT and ST tasks and boosts the performance on ST benchmarks, MuST-C and Augmented Librispeech, to a new state-of-the-art. Specifically, Chimera obtains 27.1 BLEU on MuST-C EN-DE, improving the SOTA by a +1.9 BLEU margin. Further experimental analyses demonstrate that the shared semantic space indeed conveys common knowledge between these two tasks and thus paves a new way for augmenting training resources across modalities. Code, data, and resources are available at https://github.com/Glaciohound/Chimera-ST.

Authors

Lei Li

15 papers

Heng Ji

11 papers

Mingxuan Wang

4 papers

TL;DR

Abstract

Authors

References64 items

Listen, Understand and Translate: Triple Supervision Decouples End-to-end Speech-to-text Translation

Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation

Bridging the Modality Gap for Speech-to-Text Translation

A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks

Adaptive Feature Selection for End-to-End Speech Translation

Consecutive Decoding for Speech-to-text Translation

TED: Triple Supervision Decouples End-to-end Speech-to-text Translation

Language-aware Interlingua for Multilingual Neural Machine Translation

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Self-Training for End-to-End Speech Translation

Jointly Trained Transformers Models for Spoken Language Translation

Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation

Curriculum Pre-training for End-to-End Speech Translation

ESPnet-ST: All-in-One Speech Translation Toolkit

A Comparative Study on End-to-End Speech to Text Translation

On Using SpecAugment for End-to-End Speech Translation

Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learning

Neural Speech Translation using Lattice Transformations and Graph Networks

Instance-based Model Adaptation for Direct Speech Translation

Analyzing ASR Pretraining for Low-Resource Speech-to-Text Translation

Multilingual End-to-End Speech Translation

Breaking the Data Barrier: Towards Robust Speech Translation via Adversarial Stability Training

Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation

Adapting Transformer to End-to-End Spoken Language Translation

Harnessing Indirect Training Data for End-to-End Automatic Speech Translation: Tricks of the Trade

Enhancing Transformer for End-to-end Speech-to-Text Translation

Lattice Transformer for Speech Translation

Self-Attentional Models for Lattice Inputs

Fluent Translations from Disfluent Speech in End-to-End Speech Translation

MuST-C: a Multilingual Speech Translation Corpus

End-to-End Speech Translation with Knowledge Distillation

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax

The USTC-NEL Speech Translation system at IWSLT 2018

End-to-End Speech Translation with the Transformer

Towards Fluent Translations From Disfluent Speech

Leveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation

Multilingual NMT with a Language-Independent Attention Bridge

Pre-training on high-resource speech recognition improves low-resource speech-to-text translation

Multilingual Seq2seq Training with Similarity Loss for Cross-Lingual Document Classification

Towards Robust Neural Machine Translation

A neural interlingua for multilingual machine translation

End-to-End Automatic Speech Translation of Audiobooks

Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluation

Attention is All you Need

Neural Lattice-to-Sequence Models for Uncertain Inputs

Sequence-to-Sequence Models Can Directly Translate Foreign Speech

Findings of the 2016 Conference on Machine Translation

An Attentional Model for Speech Translation Without Transcription

OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles

Findings of the 2014 Workshop on Statistical Machine Translation

Reconstructing False Start Errors in Spontaneous Speech Text

Reading Differences and Brain: Cortical Integration of Speech and Print in Sentence Processing Varies With Reader Skill

Converging Language Streams in the Human Temporal Lobe

Integration of Letters and Speech Sounds in the Human Brain

Neural Correlates of Lexical Access during Visual Word Recognition

Speech production: Wernicke, Broca and beyond.

Bleu: a Method for Automatic Evaluation of Machine Translation

Spoken Language Translation (IWSLT

The IWSLT 2018 Evaluation Campaign

Japan, May 7-12,

Translate : A Proof of Concept for End-to-End Speech-to-Text Translation

A System for Simultaneous Translation of Lectures and Speeches

Automatic sentence segmentation and punctuation prediction for spoken language translation

Field of Study

Venue Information

Name

Type

URL