Conformal prediction for text infilling and part-of-speech prediction (2021-11-04T00:00:00.000000Z)

TL;DR

This paper proposes inductive conformal prediction (ICP) algorithms for the tasks of text infilling and part-of-speech (POS) prediction for natural language data and designs new ICP-enhanced algorithms for POS tagging based on BERT and BiLSTM models.

Abstract

Modern machine learning algorithms are capable of providing remarkably accurate point-predictions; however, questions remain about their statistical reliability. Unlike conventional machine learning methods, conformal prediction algorithms return confidence sets (i.e., set-valued predictions) that correspond to a given significance level. Moreover, these confidence sets are valid in the sense that they guarantee finite sample control over type 1 error probabilities, allowing the practitioner to choose an acceptable error rate. In our paper, we propose inductive conformal prediction (ICP) algorithms for the tasks of text infilling and part-of-speech (POS) prediction for natural language data. We construct new ICP-enhanced algorithms for POS tagging based on BERT (bidirectional encoder representations from transformers) and BiLSTM (bidirectional long short-term memory) models. For text infilling, we design a new ICP-enhanced BERT algorithm. We analyze the performance of the algorithms in simulations using the Brown Corpus, which contains over 57,000 sentences. Our results demonstrate that the ICP algorithms are able to produce valid set-valued predictions that are small enough to be applicable in real-world applications. We also provide a real data example for how our proposed set-valued predictions can improve machine generated audio transcriptions.

Authors

N. Dey

1 papers

Jing Ding

1 papers

Jack G. Ferrell

1 papers

TL;DR

Abstract

Authors

References73 items

Well-calibrated confidence measures for multi-label text classification with a large number of labels

Few-shot Conformal Prediction with Auxiliary Tasks

Reliable Part-of-Speech Tagging of Historical Corpora through Set-Valued Prediction

Relaxed Conformal Prediction Cascades for Efficient Inference Over Many Labels

Deep Conformal Prediction for Robust Models

Enabling Language Models to Fill in the Blanks

Knowing what You Know: valid and validated confidence sets in multiclass and multilabel prediction

A Primer in BERTology: What We Know About How BERT Works

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Blank Language Models

E-mail Address

A Deep Neural Network Conformal Predictor for Multi-label Text Classification

On the Variance of the Adaptive Learning Rate and Beyond

Efficient set-valued prediction in multi-class classification

Efficient Algorithms for Set-Valued Prediction in Multi-Class Classification

TIGS: An Inference Algorithm for Text Infilling with Gradient Search

ERNIE: Enhanced Language Representation with Informative Entities

Learning Better Internal Structure of Words for Sequence Labeling

A Deep Recurrent Neural Network with BiLSTM model for Sentiment Classification

Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings

TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A Simple and Effective Approach to the Story Cloze Test

Deep Contextualized Word Representations

MaskGAN: Better Text Generation via Filling in the ______

Overcoming the vanishing gradient problem in plain recurrent networks

Robust Multilingual Part-of-Speech Tagging via Adversarial Training

Empower Sequence Labeling with Task-Aware Neural Language Model

Attention is All you Need

Semi-supervised sequence tagging with bidirectional language models

Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss

Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Recurrent Neural Network

A Primer on Neural Network Models for Natural Language Processing

Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

Advances in natural language processing

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Structure Regularization for Structured Prediction

GloVe: Global Vectors for Word Representation

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

A Clockwork RNN

Understanding the exploding gradient problem

Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics?

Inductive Conformal Prediction: Theory and Application to Neural Networks

A tutorial on conformal prediction

Algorithmic Learning in a Random World

Learning precise timing with lstm recurrent networks

Learning Precise Timing with LSTM Recurrent Networks

Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

TnT – A Statistical Part-of-Speech Tagger

Machine-Learning Applications of Algorithmic Randomness

The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions

Long Short-Term Memory

Learning long-term dependencies with gradient descent is difficult

Robust part-of-speech tagging using a hidden Markov model

Finding Structure in Time

Address

Efficient Conformal Prediction via Cascaded Inference with Expanded Admission

Transformer-based conformal predictors for paraphrase detection

BERT-based conformal predictor for sentiment analysis

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Text inﬁlling

Conformal Prediction for Text Infilling and Part-of-Speech Prediction 83 I. Improving language understanding by generative pre-training (2018)

Improving Language Understanding by Generative Pre-Training

2018 Conference of the North American Chapter of the

A corpus and evaluation framework for deeper understanding of commonsense stories

Understanding LSTM networks

Advances in Neural Information Processing Systems 27

English Giga-word ﬁfth edition. Technical report, Technical Report

English Gigaword fifth edition

Speech and language processing: an introduction to natural language processing

Conformal prediction for text inﬁlling and part-of-speech prediction 13

Brown Corpus manual

North Carolina State University, United States

Field of Study

Journal Information

Name