On the Utility of Self-Supervised Models for Prosody-Related Tasks (2022-10-13T00:00:00.000000Z)

TL;DR

A new evaluation framework, “SUPERB-prosody,” is presented, consisting of three prosody-related downstream tasks and two pseudo tasks, which concludes that SSL speech models are highly effective for prosodic-related tasks.

Abstract

Self-Supervised Learning (SSL) from speech data has produced models that have achieved remarkable performance in many tasks, and that are known to implicitly represent many aspects of information latently present in speech signals. However, relatively little is known about the suitability of such models for prosody-related tasks or the extent to which they encode prosodic information. We present a new evaluation framework, “SUPERB-prosody,” consisting of three prosody-related downstream tasks and two pseudo tasks. We find that 13 of the 15 SSL models outperformed the baseline on all the prosody-related tasks. We also show good performance on two pseudo tasks: prosody reconstruction and future prosody prediction. We further analyze the layerwise contributions of the SSL models. Overall we conclude that SSL speech models are highly effective for prosody-related tasks. We release our code11https://github.com/JSALT-2022-SSL/superb-prosody for the community to support further investigation of SSL models' utility for prosody.

Authors

Hung-yi Lee

13 papers

Guan-Ting Lin

1 papers

Chiyu Feng

1 papers

TL;DR

Abstract

Authors

References38 items

Comparison of Models for Detecting Off-Putting Speaking Styles

On Combining Global and Localized Self-Supervised Models of Speech

Self-Supervised Speech Representation Learning: A Review

SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities

A multimodal fusion method for sarcasm detection based on late fusion

SLUE: New Benchmark Tasks For Spoken Language Understanding Evaluation on Natural Speech

WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

Distilhubert: Speech Representation Learning by Layer-Wise Distillation of Hidden-Unit Bert

Learning de-identified representations of prosody from raw audio

Layer-Wise Analysis of a Self-Supervised Speech Representation Model

Compiling REAPER (Robust Epoch And Pitch EstimatoR)

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

SUPERB: Speech processing Universal PERformance Benchmark

What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure

Spoken Language Interaction with Robots: Research Issues and Recommendations, Report from the NSF Future Directions Workshop

Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies

MLS: A Large-Scale Multilingual Dataset for Speech Research

Similarity Analysis of Self-Supervised Speech Representations

AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines

TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Vector-Quantized Autoregressive Predictive Coding

Longformer: The Long-Document Transformer

Towards Learning a Universal Non-Semantic Representation of Speech

Unsupervised Pretraining Transfers Well Across Languages

Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders

vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)

LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech

An Unsupervised Autoregressive Model for Speech Representation Learning

Prosodic Patterns in English Conversation

Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Deep multimodal fusion for persuasiveness prediction

Computational Analysis of Persuasiveness in Social Multimedia: A Novel Dataset and Multimodal Prediction Approach

Speech recognition

“Stepped intonation contours: A new ﬁeld of complexity,”

Comparison of models

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names