Token-Level Supervised Contrastive Learning for Punctuation Restoration (2021-07-19T00:00:00.000000Z)

TL;DR

A token-level supervised contrastive learning method that aims at maximizing the distance of representation of different punctuation marks in the embedding space is proposed that obtains up to 3.2% absolute F1 improvement on the test set.

Abstract

Punctuation is critical in understanding natural language text. Currently, most automatic speech recognition (ASR) systems do not generate punctuation, which affects the performance of downstream tasks, such as intent detection and slot filling. This gives rise to the need for punctuation restoration. Recent work in punctuation restoration heavily utilizes pre-trained language models without considering data imbalance when predicting punctuation classes. In this work, we address this problem by proposing a token-level supervised contrastive learning method that aims at maximizing the distance of representation of different punctuation marks in the embedding space. The result shows that training with token-level supervised contrastive learning obtains up to 3.2% absolute F1 improvement on the test set.

Authors

Xubo Liu

5 papers

Qiushi Huang

4 papers

Tom Ko

7 papers

TL;DR

Abstract

Authors

References33 items

Automatic punctuation restoration with BERT models

Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning

Focal Loss for Punctuation Prediction

Joint Prediction of Punctuation and Disfluency in Speech Transcripts

Efficient Automatic Punctuation Restoration Using Bidirectional Transformers with Robust Inference

Supervised Contrastive Learning

Adversarial Transfer Learning for Punctuation Restoration

Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection

Transfer Learning for Punctuation Prediction

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Leveraging a Character, Word and Prosody Triplet for an ASR Error Robust and Agglutination Friendly Punctuation Approach

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punctuation Restoration

Self-attention Based Model for Punctuation Prediction Using Word and Speech Embeddings

Empirical Evaluation and Combination of Punctuation Prediction Models Applied to Broadcast News

Representation Learning with Contrastive Predictive Coding

Punctuation Prediction Model for Conversational Speech

Decoupled Weight Decay Regularization

Distilling Knowledge from an Ensemble of Models for Punctuation Prediction

Attention is All you Need

TristouNet: Triplet loss for speaker turn embedding

Bidirectional Recurrent Neural Network with Attention Mechanism for Punctuation Restoration

Punctuation Prediction for Unsegmented Transcript Based on Word Vector

FaceNet: A unified embedding for face recognition and clustering

Better Punctuation Prediction with Dynamic Conditional Random Fields

Restoring punctuation and capitalization in transcribed speech

Enriching speech recognition with automatic detection of sentence boundaries and disfluencies

A combined punctuation generation and speech recognition system and its performance enhancement using prosody

Cyberpunc: a lightweight punctuation annotation system for speech

Punctuation Restoration using Transformer Models for High-and Low-Resource Languages

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Improved models for automatic punctuation prediction for spoken and written text

Punctuation annotation using statistical prosody models.

Field of Study

Venue Information

Name

Type

URL

Alternate Names