Claim Check-Worthiness Detection as Positive Unlabelled Learning (2020-03-05T00:00:00.000000Z)

TL;DR

The best performing method is a unified approach which automatically corrects for this using a variant of positive unlabelled learning that finds instances which were incorrectly labelled as not check-worthy.

Abstract

As the first step of automatic fact checking, claim check-worthiness detection is a critical component of fact checking systems. There are multiple lines of research which study this problem: check-worthiness ranking from political speeches and debates, rumour detection on Twitter, and citation needed detection from Wikipedia. To date, there has been no structured comparison of these various tasks to understand their relatedness, and no investigation into whether or not a unified approach to all of them is achievable. In this work, we illuminate a central challenge in claim check-worthiness detection underlying all of these tasks, being that they hinge upon detecting both how factual a sentence is, as well as how likely a sentence is to be believed without verification. As such, annotators only mark those instances they judge to be clear-cut check-worthy. Our best performing method is a unified approach which automatically corrects for this using a variant of positive unlabelled learning that finds instances which were incorrectly labelled as not check-worthy. In applying this, we out-perform the state of the art in two of the three tasks studied for claim check-worthiness detection in English.

Authors

Isabelle Augenstein

17 papers

Dustin Wright

2 papers

TL;DR

Abstract

Authors

References45 items

Generating Label Cohesive and Well-Formed Adversarial Claims

Time-Aware Evidence Ranking for Fact-Checking

Multi-Hop Fact Checking of Political Claims

Generating Fact Checking Explanations

CheckThat! at CLEF 2020: Enabling the Automatic Identification and Verification of Claims in Social Media

A Graph-Theoretic Embedding-Based Approach for Rumor Detection in Twitter

Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of Claims

MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims

Distantly Supervised Named Entity Recognition using Positive-Unlabeled Learning

Detect Rumors on Twitter by Promoting Information Campaigns with Generative Adversarial Learning

Neural Check-Worthiness Ranking with Weak Supervision: Finding Sentences for Fact-Checking

Citation Needed: A Taxonomy and Algorithmic Assessment of Wikipedia's Verifiability

Learning from positive and unlabeled data: a survey

Towards Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection

Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthiness

Automated Fact Checking: Task Formulations, Methods and Future Directions

ClaimRank: Detecting Check-Worthy Claims in Arabic and English

Universal Language Model Fine-tuning for Text Classification

Discourse-aware rumour stance classification in social media using sequential classifiers

Exploiting Context for Rumour Detection in Social Media

ClaimBuster: The First-ever End-to-end Fact-checking System

Attention is All you Need

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

Positive-Unlabeled Learning with Non-Negative Risk Estimator

The rise of fact-checking sites in Europe

Supervised Keyphrase Extraction as Positive Unlabeled Learning

Enriching Word Vectors with Subword Information

The spreading of misinformation online

Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads

Spotting Fake Reviews via Collective Positive-Unlabeled Learning

Analysis of Learning from Positive and Unlabeled Data

Positive Unlabeled Learning for Deceptive Reviews Detection

Fact Checking: Task definition and dataset construction

Learning classifiers from only positive and unlabeled data

Learning from positive and unlabeled examples

Positive and Unlabeled Examples Help Learning

PAC Learning from Positive Statistical Queries

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Neural Weakly Supervised Fact Check-Worthiness Detection with Contrastive Sampling-Based Ranking Loss

A Context-Aware Approach for Detecting Worth-Checking Claims in Political Debates

Digital Wildﬁres in a Hyper-connected World

A Examples of PUC Improvements for Rumour Detection Examples of improvements for rumour detection using PUC can be found in Table 5

B Reproducibility B.1 Computing Infrastructure All experiments were run on a shared Re-quested jobs consisted of 16GB of

NVIDIA Titan X GPU with 12GB of RAM. B.2 Average Runtimes

Veriﬁca-tion of Political Claims. Task 1: Check-Worthiness

Field of Study

Venue Information

Name

Type

URL