Investigating the Robustness of Modelling Decisions for Few-Shot Cross-Topic Stance Detection: A Preregistered Study (2024-04-05T00:00:00.000000Z)

TL;DR

The robustness of operationalization choices for few-shot stance detection is investigated, with special attention to modelling stance across different topics, and cross-encoding out-performs bi-encoding and adding NLI training to models gives considerable improvement.

Abstract

For a viewpoint-diverse news recommender, identifying whether two news articles express the same viewpoint is essential. One way to determine “same or different” viewpoint is stance detection. In this paper, we investigate the robustness of operationalization choices for few-shot stance detection, with special attention to modelling stance across different topics. Our experiments test pre-registered hypotheses on stance detection. Specifically, we compare two stance task definitions (Pro/Con versus Same Side Stance), two LLM architectures (bi-encoding versus cross-encoding), and adding Natural Language Inference knowledge, with pre-trained RoBERTa models trained with shots of 100 examples from 7 different stance detection datasets. Some of our hypotheses and claims from earlier work can be confirmed, while others give more inconsistent results. The effect of the Same Side Stance definition on performance differs per dataset and is influenced by other modelling choices. We found no relationship between the number of training topics in the training shots and performance. In general, cross-encoding out-performs bi-encoding, and adding NLI training to our models gives considerable improvement, but these results are not consistent across all datasets. Our results indicate that it is essential to include multiple datasets and systematic modelling experiments when aiming to find robust modelling choices for the concept ‘stance’.

Authors

Myrthe Reuver

1 papers

Suzan Verberne

1 papers

Antske Fokkens

1 papers

TL;DR

Abstract

Authors

References59 items

Dive into the Chasm: Probing the Gap between In- and Cross-Topic Generalization

Surveying (Dis)Parities and Concerns of Compute Hungry NLP Research

Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection

A Two-Sided Discussion of Preregistration of NLP Research

Topic Ontologies for Arguments

Robust Integration of Contextual Information for Cross-Target Stance Detection

Is my stance the same as your stance? A cross validation study of stance detection datasets

Efficient Few-Shot Learning Without Prompts

Will It Blend? Mixing Training Paradigms & Prompting for Argument Quality Prediction

Nudging towards news diversity: A theoretical framework for facilitating diverse news consumption through recommender design

FETA: A Benchmark for Few-Sample Task Transfer in Open-Domain Dialogue

Towards Climate Awareness in NLP Research

Cluster & Tune: Boost Cold Start Performance in Text Classification

Towards Analyzing the Bias of News Recommender Systems Using Sentiment and Stance Detection

Is Stance Detection Topic-Independent and Cross-topic Generalizable? - A Reproduction Study

Tribrid: Stance Classification with Neural Inconsistency Detection

(Mis)alignment Between Stance Expressed in Social Media Data and Public Opinion Surveys

Adversarial Learning for Zero-Shot Stance Detection on Social Media

Cross-Domain Label-Adaptive Stance Detection

Preregistering NLP research

Operationalizing Framing to Support Multiperspective Recommendations of Opinion Pieces

Transformers: State-of-the-Art Natural Language Processing

COVIDLies: Detecting COVID-19 Misinformation on Social Media

On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines

Exploring and Predicting Transferability across NLP Tasks

Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?

Same Side Stance Classification Task: Facilitating Argument Stance Classification by Fine-tuning a BERT Model

Stance Detection Benchmark: How Robust is Your Stance Detection?

Adversarial NLI: A New Benchmark for Natural Language Understanding

STANCY: Stance Classification Based on Consistency Cues

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Modeling Transferable Topics for Cross-Target Stance Detection

On the Democratic Role of News Recommenders

Seeing Things from a Different Angle:Discovering Diverse Perspectives about Claims

Classification and Clustering of Arguments with Contextualized Word Embeddings

Cross-Target Stance Classification with Self-Attention Networks

Cross-topic Argument Mining from Heterogeneous Sources

The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Ethical Considerations in NLP Shared Tasks

Stance Classification of Context-Dependent Claims

SemEval-2016 Task 6: Detecting Stance in Tweets

A large annotated corpus for learning natural language inference

Stance Classification of Ideological Debates: Data, Models, Features, and Constraints

A Corpus for Research on Deliberation and Debate

The stance triangle

Get out the vote: Determining support or opposition from Congressional floor-debate transcripts

Why does it work

Dynamic Stance: Modeling Discussions by Labeling the Interactions

Detecting Dissonant Stance in Social Media: The Role of Topic Exposure

Overview of the 2022 Validity and Novelty Prediction Shared Task

On Classifying whether Two Texts are on the Same Side of an Argument

Spurious Correlations in Cross-Topic Argument Mining

Enhancing Zero-shot and Few-shot Stance Detection with Commonsense Knowledge Graph

Stance detection: A survey

Same Side Stance Classification

Siamese Neural Network for Same Side Stance Classification

The# benderrule: On naming the languages we study and why it matters

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names