Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech

Published in

Interspeech(2018)

External Links:

Generate Graph DownloadPDF

TL;DR

The proposed Speech2Vec model, a novel deep neural network architecture for learning fixed-length vector representations of audio segments excised from a speech corpus, is based on a RNN Encoder-Decoder framework, and borrows the methodology of skipgrams or continuous bag-of-words for training.

Abstract

In this paper, we propose a novel deep neural network architecture, Speech2Vec, for learning fixed-length vector representations of audio segments excised from a speech corpus, where the vectors contain semantic information pertaining to the underlying spoken words, and are close to other vectors in the embedding space if their corresponding underlying spoken words are semantically similar. The proposed model can be viewed as a speech version of Word2Vec. Its design is based on a RNN Encoder-Decoder framework, and borrows the methodology of skipgrams or continuous bag-of-words for training. Learning word embeddings directly from speech enables Speech2Vec to make use of the semantic information carried by speech that does not exist in plain text. The learned word embeddings are evaluated and analyzed on 13 widely used word similarity benchmarks, and outperform word embeddings learned by Word2Vec from the transcriptions.

Authors

James R. Glass

16 papers

Yu-An Chung

7 papers

References48 items

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

Supervised and Unsupervised Transfer Learning for Question Answering

Learning Word Embeddings from Speech

Automatic differentiation in PyTorch

Speech-Based Visual Question Answering

Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech

Published in

Interspeech(2018)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

James R. Glass

16 papers

Yu-An Chung

7 papers

References48 items

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

Supervised and Unsupervised Transfer Learning for Question Answering

Learning Word Embeddings from Speech

Automatic differentiation in PyTorch

Speech-Based Visual Question Answering

Character Composition Model with Convolutional Neural Networks for Dependency Parsing on Morphologically Rich Languages

Neural AMR: Sequence-to-Sequence Models for Parsing and Generation

An embedded segmental K-means model for unsupervised segmentation and clustering of speech

Learning Word-Like Units from Joint Audio-Visual Analysis

Discriminative acoustic word embeddings: Tecurrent neural network-based approaches

Multi-view Recurrent Neural Acoustic Word Embeddings

Towards Machine Comprehension of Spoken Content: Initial TOEFL Listening Comprehension Test by Machine

SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity

Enriching Word Vectors with Subword Information

A segmental framework for fully-unsupervised large-vocabulary speech recognition

Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss

Neural Architectures for Named Entity Recognition

Audio Word2Vec: Unsupervised Learning of Audio Segment Representations Using Sequence-to-Sequence Autoencoder

Deep multimodal semantic embeddings for speech and images

Deep convolutional acoustic word embeddings using word-pair side information

Evaluation methods for unsupervised word embeddings

Character-Aware Neural Language Models

Effective Approaches to Attention-based Neural Machine Translation

Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs

Sequence to Sequence -- Video to Text

Librispeech: An ASR corpus based on public domain audio books

GloVe: Global Vectors for Word Representation

An Unsupervised Model for Instance Level Subcategorization Acquisition

Sequence to Sequence Learning with Neural Networks

Neural Machine Translation by Jointly Learning to Align and Translate

SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Community Evaluation and Exchange of Word Vectors at wordvectors.org

Fixed-dimensional acoustic embeddings of variable-length segments in low-resource settings

Distributed Representations of Words and Phrases and their Compositionality

Better Word Representations with Recursive Neural Networks for Morphology

Large-scale learning of word relatedness with constraints

Distributional Semantics in Technicolor

A word at a time: computing word relatedness using temporal semantic analysis

A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches

Long Short-Term Memory

Research Design and Statistical Analysis

Contextual correlates of synonymy

Unsupervised Learning of Spoken Language with Visual Context

Word embeddings for speech recognition

Visualizing Data using t-SNE

Verb similarity on the taxonomy of WordNet

Contextual correlates of semantic similarity

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

Interspeech

Type

conference

URL

https://www.isca-speech.org/iscaweb/index.php/conferences/interspeech

Alternate Names

Conf Int Speech Commun Assoc
INTERSPEECH
Conference of the International Speech Communication Association