Foreseeing the Benefits of Incidental Supervision (2020-06-09T00:00:00.000000Z)

TL;DR

A unified PAC-Bayesian motivated informativeness measure, PABI, is proposed that characterizes the uncertainty reduction provided by incidental supervision signals, and its effectiveness is demonstrated by quantifying the value added by various types of incidental signals to sequence tagging tasks.

Abstract

Real-world applications often require improved models by leveraging *a range of cheap incidental supervision signals*. These could include partial labels, noisy labels, knowledge-based constraints, and cross-domain or cross-task annotations – all having statistical associations with gold annotations but not exactly the same. However, we currently lack a principled way to measure the benefits of these signals to a given target task, and the common practice of evaluating these benefits is through exhaustive experiments with various models and hyperparameters. This paper studies whether we can, *in a single framework, quantify the benefits of various types of incidental signals for a given target task without going through combinatorial experiments*. We propose a unified PAC-Bayesian motivated informativeness measure, PABI, that characterizes the uncertainty reduction provided by incidental supervision signals. We demonstrate PABI’s effectiveness by quantifying the value added by various types of incidental signals to sequence tagging tasks. Experiments on named entity recognition (NER) and question answering (QA) show that PABI’s predictions correlate well with learning performance, providing a promising way to determine, ahead of learning, which supervision signals would be beneficial.

Authors

Mingyuan Zhang

6 papers

Dan Roth

19 papers

Qiang Ning

7 papers

TL;DR

Abstract

Authors

References76 items

Foundations of Machine Learning

Transformers: State-of-the-Art Natural Language Processing

Learnability with Indirect Supervision Signals

UnifiedQA: Crossing Format Boundaries With a Single QA System

Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

Frosting Weights for Better Continual Training

Named Entity Recognition with Partially Annotated Training Data

QuASE: Question-Answer Driven Sentence Encoding

Partial Or Complete, That’s The Question

Linguistic Knowledge and Transferability of Contextual Representations

Revisiting Self-Supervised Visual Representation Learning

Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks

Localized Structured Prediction

Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces

Large-Scale QA-SRL Parsing

Zero-Shot Relation Extraction via Reading Comprehension

Deep Learning is Robust to Massive Label Noise

Will my auxiliary tagging task help? Estimating Auxiliary Tasks Effectivity in Multi-Task Learning

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

Incidental Supervision: Moving beyond Supervised Learning

Generalisation in named entity recognition: A quantitative analysis

Results of the WNUT16 Named Entity Recognition Shared Task

NewsQA: A Machine Comprehension Dataset

SQuAD: 100,000+ Questions for Machine Comprehension of Text

A Consistent Regularization Approach for Structured Prediction

Adam: A Method for Stochastic Optimization

GloVe: Global Vectors for Word Representation

Learnability of the Superset Label Learning Problem

Learning with Noisy Labels

Proper losses for learning from partial labels

Structured learning with constrained conditional models

Exploiting Partial Annotations with EM Training

Learning from Partial Labels

A Survey on Transfer Learning

A theory of learning from different domains

A discriminative model for semi-supervised learning

A new learning paradigm: Learning using privileged information

A notion of task relatedness yielding provable multiple-task learning guarantees

Learning and Inference with Constraints

Transfer Learning using Kolmogorov Complexity: Basic Theory and Empirical Evaluations

PAC-BAYESIAN SUPERVISED CLASSIFICATION: The Thermodynamics of Statistical Learning

From ɛ-entropy to KL-entropy: Analysis of minimum information complexity density estimation

Estimating relatedness via data compression

OntoNotes: The 90% Solution

Convexity, Classification, and Risk Bounds

Proof of Theorem 2

A Note on the PAC Bayesian Theorem

A Linear Programming Formulation for Global Inference in Natural Language Tasks

Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

PAC-Bayesian Stochastic Model Selection

PAC-Bayesian Generalisation Error Bounds for Gaussian Process Classification

Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

PAC-Bayesian model averaging

Some PAC-Bayesian Theorems

Clustering Learning Tasks and the Selective Cross-Task Transfer of Knowledge

Text Chunking using Transformation-Based Learning

Hints and the VC Dimension

Learning From Noisy Examples

Learning from Noisy Labels with No Change to the Training Process

Transfer Learning via Minimizing the Performance Gap Between Domains

Different algorithms make different assumptions on the concept class. For example, SVM aims to find the maximum-margin hyperplane (see more in Chapter

cS is a consistent concept with S returned by A

A Theory of Learning with Corrupted Labels

Crowdsourcing Question-Answer Meaning Representations

Augenstein et al., 2017)

Stability and Generalization in Structured Prediction

The Groningen Meaning Bank

Our theorem is based on binary classification mainly because the error item in Eq

Guiding Semi-Supervision with Constraint-Driven Learning

Simplified PAC-Bayesian Margin Bounds

NER with mixed inductive signals. A more

Towards a theory of Coherent Concepts

Minimum Message Length and Kolmogorov Complexity

Theoretical Models of Learning to Learn

A theory of the learnable

Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities