Word Discovery in Visually Grounded, Self-Supervised Speech Models (2022-03-28T00:00:00.000000Z)

TL;DR

It is shown that powerful word segmentation and clustering capability emerges within the model's self-attention heads, suggesting that the visual grounding task is a crucial component of the word discovery capability the authors observe.

Abstract

We present a method for visually-grounded spoken term discovery. After training either a HuBERT or wav2vec2.0 model to associate spoken captions with natural images, we show that powerful word segmentation and clustering capability emerges within the model's self-attention heads. Our experiments reveal that this ability is not present to nearly the same extent in the base HuBERT and wav2vec2.0 models, suggesting that the visual grounding task is a crucial component of the word discovery capability we observe. We also evaluate our method on the Buckeye word segmentation and ZeroSpeech spoken term discovery tasks, where we perform on par with or better than currently published methods on several metrics. Code and model weights are available at https://github.com/jasonppy/word-discovery.

Authors

Puyuan Peng

1 papers

David Harwath

1 papers

TL;DR

Abstract

Authors

References45 items

Learning English with Peppa Pig

Word Segmentation on Discovered Phone Units With Dynamic Programming and Self-Supervised Scoring

Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling

A Translation Framework for Visually Grounded Spoken Unit Discovery

Contrastive Prediction Strategies for Unsupervised Segmentation and Categorization of Phonemes and Words

Fast-Slow Transformer for Visually Grounding Speech

Layer-Wise Analysis of a Self-Supervised Speech Representation Model

Attention-Based Keyword Localisation in Speech using Visual Grounding

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

Cross-Modal Discrete Representation Learning

Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation

Emerging Properties in Self-Supervised Vision Transformers

Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques

Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? - A computational investigation

Text-Free Image-to-Speech Synthesis Using Learned Segmental Units

Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networks

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Unsupervised Discovery of Recurring Speech Patterns Using Probabilistic Adaptive Metrics

Self-Expressing Autoencoders for Unsupervised Spoken Term Discovery

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Learning to Recognise Words Using Visually Grounded Speech

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech

Large-Scale Representation Learning from Visually Grounded Untranscribed Speech

Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech

Transfer Learning from Audio-Visual Grounding to Speech Recognition

Representation Learning with Contrastive Predictive Coding

Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input

The zero resource speech challenge 2017

Attention is All you Need

An embedded segmental K-means model for unsupervised segmentation and clustering of speech

Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner

A segmental framework for fully-unsupervised large-vocabulary speech recognition

Deep multimodal semantic embeddings for speech and images

Unsupervised Lexicon Discovery from Acoustic Input

Learning Words from Images and Speech

Deep visual-semantic alignments for generating image descriptions

Microsoft COCO: Common Objects in Context

A summary of the 2012 JHU CLSP workshop on zero resource speech technologies and models of early language acquisition

Efficient spoken term discovery using randomized algorithms

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Unsupervised Learning of Spoken Language with Visual Context

Unsupervised word discovery from speech using automatic segmentation into syllable-like units

Towards spoken term discovery at scale with zero resources

Unsupervised Pattern Discovery in Speech

The Buckeye corpus of conversational speech: labeling conventions and a test of transcriber reliability

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names