DynaSent: A Dyn… (2020-12-30T00:00:00.000000Z)

TL;DR

DynaSent (‘Dynamic Sentiment’), a new English-language benchmark task for ternary (positive/negative/neutral) sentiment analysis, is introduced and a report on the dataset creation effort is reported, focusing on the steps taken to increase quality and reduce artifacts.

Abstract

We introduce DynaSent (‘Dynamic Sentiment’), a new English-language benchmark task for ternary (positive/negative/neutral) sentiment analysis. DynaSent combines naturally occurring sentences with sentences created using the open-source Dynabench Platform, which facilities human-and-model-in-the-loop dataset creation. DynaSent has a total of 121,634 sentences, each validated by five crowdworkers, and its development and test splits are designed to produce chance performance for even the best models we have been able to develop; when future models solve this task, we will use them to create DynaSent version 2, continuing the dynamic evolution of this benchmark. Here, we report on the dataset creation effort, focusing on the steps we took to increase quality and reduce artifacts. We also present evidence that DynaSent’s Neutral category is more coherent than the comparable category in other benchmarks, and we motivate training models from scratch for each round over successive fine-tuning.

Authors

Christopher Potts

12 papers

Douwe Kiela

11 papers

Zhengxuan Wu

5 papers

References78 items

Transformers: State-of-the-Art Natural Language Processing

Towards Ecologically Valid Research on Language User Interfaces

How Can We Accelerate Progress Towards Human-like Linguistic Generalization?

Sentiment

Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension

TL;DR

Abstract

Authors

Christopher Potts

12 papers

Douwe Kiela

11 papers

Zhengxuan Wu

5 papers

TL;DR

Abstract

Authors

References78 items

Transformers: State-of-the-Art Natural Language Processing

Towards Ecologically Valid Research on Language User Interfaces

How Can We Accelerate Progress Towards Human-like Linguistic Generalization?

Sentiment

Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension

TL;DR

Abstract

Authors

References78 items

Transformers: State-of-the-Art Natural Language Processing

Towards Ecologically Valid Research on Language User Interfaces

How Can We Accelerate Progress Towards Human-like Linguistic Generalization?

Sentiment

Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension

Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects

Inherent Disagreements in Human Textual Inferences

Adversarial NLI: A New Benchmark for Natural Language Understanding

Universal Adversarial Triggers for Attacking and Analyzing NLP

Towards Debiasing Fact Verification Models

Adversarial Attack on Sentiment Classification

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference

The Risk of Racial Bias in Hate Speech Detection

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments

HellaSwag: Can a Machine Really Finish Your Sentence?

Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets

DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey

Analyzing Compositionality-Sensitivity of NLI Models

Model Cards for Model Reporting

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks

Stress Test Evaluation for Natural Language Inference

Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems

Breaking NLI Systems with Sentences that Require Simple Lexical Inferences

Hypothesis Only Baselines in Natural Language Inference

Gender Bias in Coreference Resolution

Generating Natural Language Adversarial Examples

Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment

Datasheets for datasets

Annotation Artifacts in Natural Language Inference Data

Using Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks

SemEval-2017 Task 4: Sentiment Analysis in Twitter

Adversarial Examples for Evaluating Reading Comprehension Systems

Attention is All you Need

Social Bias in Elicited Natural Language Inferences

Character-level Convolutional Networks for Text Classification

Neural Machine Translation of Rare Words with Subword Units

On our best behaviour

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Norms of valence, arousal, and dominance for 13,915 English lemmas

Learning Attitudes and Attributes from Multi-aspect Reviews

Did It Happen? The Pragmatic Complexity of Veridicality Assessment

Sentiment Analysis and Opinion Mining

Learning Word Vectors for Sentiment Analysis

SentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining

Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks

Opinion spam and analysis

Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis

Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales

Mining and summarizing customer reviews

A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts

Word frequency distributions

Maximum Likelihood Estimation of Observer Error‐Rates Using the EM Algorithm

Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper

A computer approach to content analysis: studies using the General Inquirer system

Negative Side Effects and AI Agent Indicators: Experiments in SafeLife

Emotion AI has great promise (when used responsibly)

Is BERT Really Robust? Natural Language Attack on Text Classiﬁcation and Entailment

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

ELECTRA: Pretraining text encoders as discriminators rather than generators

CODAH: An adversarially-authored question answering dataset for common sense

Darling or Babygirl ? Investigating Stylistic Bias in Sentiment Analysis

Opinion Mining and Sentiment Analysis

Truth Inference in Crowdsourcing: Is the Problem Solved?