Adversarial NLI: A New Benchmark for Natural Language Understanding (2019-10-31T00:00:00.000000Z)

TL;DR

This work introduces a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure, and shows that non-expert annotators are successful at finding their weaknesses.

Abstract

We introduce a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure. We show that training models on this new dataset leads to state-of-the-art performance on a variety of popular NLI benchmarks, while posing a more difficult challenge with its new test set. Our analysis sheds light on the shortcomings of current state-of-the-art models, and shows that non-expert annotators are successful at finding their weaknesses. The data collection method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate.

Authors

Douwe Kiela

11 papers

Mohit Bansal

47 papers

Adina Williams

6 papers

TL;DR

Abstract

Authors

References69 items

Visual Question Answering: From Theory to Application

Adversarial Filters of Dataset Biases

Learning the Difference that Makes a Difference with Counterfactually-Augmented Data

Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets

Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack

Abductive Commonsense Reasoning

RoBERTa: A Robustly Optimized BERT Pretraining Approach

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study

Learning from On-Line User Feedback in Neural Question Answering on the Web

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

HellaSwag: Can a Machine Really Finish Your Sentence?

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

Multi-Task Deep Neural Networks for Natural Language Understanding

No Training Required: Exploring Random Encoders for Sentence Classification

Combining Fact Extraction and Verification with Neural Semantic Matching Networks

Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

Trick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering

Trick Me If You Can: Adversarial Writing of Trivia Challenge Questions

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks

Stress Test Evaluation for Natural Language Inference

Do CIFAR-10 Classifiers Generalize to CIFAR-10?

Breaking NLI Systems with Sentences that Require Simple Lexical Inferences

Hypothesis Only Baselines in Natural Language Inference

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment

FEVER: a Large-scale Dataset for Fact Extraction and VERification

Annotation Artifacts in Natural Language Inference Data

SentEval: An Evaluation Toolkit for Universal Sentence Representations

Mastering the Dungeon: Grounded Language Learning by Mechanical Turker Descent

Synthetic and Natural Noise Both Break Neural Machine Translation

Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task

Natural Language Inference over Interaction Space

Shortcut-Stacked Sentence Encoders for Multi-Domain Inference

A Continuously Growing Dataset of Sentential Paraphrases

Adversarial Examples for Evaluating Reading Comprehension Systems

Teaching Machines to Describe Images via Natural Language Feedback

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

Teaching Machines to Describe Images with Natural Language Feedback

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Reading Wikipedia to Answer Open-Domain Questions

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

The LAMBADA dataset: Word prediction requiring a broad discourse context

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Build It, Break It, Fix It: Contesting Secure Development

A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories

Deep Residual Learning for Image Recognition

The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations

A large annotated corpus for learning natural language inference

Exploring Nearest Neighbor Approaches for Image Captioning

VQA: Visual Question Answering

Never-Ending Learning

ImageNet Large Scale Visual Recognition Challenge

Regularization of Neural Networks using DropConnect

Multi-column deep neural networks for image classification

ImageNet: A large-scale hierarchical image database

The Tragedy of Hamlet, Prince of Denmark

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Trick Me If You Can: Human-in-the-loop Generation of Adversarial Question Answering Examples

CODAH: an adver-sarially authored question-answer dataset for common sense

Christos Christodoulopoulos, and Arpit Mittal

The Seventh PASCAL Recognizing Textual Entailment Challenge

The Sixth PASCAL Recognizing Textual Entailment Challenge

Gradient-based learning applied to document recognition

born May 1, 1894) was an Indian lawyer and politician who served as the Minister of Land Revenue for the Madras Presidency from March

2016) was a member of the Sicilian Mafia (“Cosa Nostra”) and was suspected of having been the head of the Corleonesi, a Mafia faction

1603. The Tragedy of Hamlet

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type