Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition (2022-10-08T00:00:00.000000Z)

TL;DR

This work proposes a unified causal framework to retrieve the causality from both new entity types and Other-Class and applies curriculum learning to mitigate the impact of label noise and introduce a self-adaptive weight for balancing the causal effects between new entities types and other-Class.

Abstract

Continual Learning for Named Entity Recognition (CL-NER) aims to learn a growing number of entity types over time from a stream of data. However, simply learning Other-Class in the same way as new entity types amplifies the catastrophic forgetting and leads to a substantial performance drop. The main cause behind this is that Other-Class samples usually contain old entity types, and the old knowledge in these Other-Class samples is not preserved properly. Thanks to the causal inference, we identify that the forgetting is caused by the missing causal effect from the old data.To this end, we propose a unified causal framework to retrieve the causality from both new entity types and Other-Class.Furthermore, we apply curriculum learning to mitigate the impact of label noise and introduce a self-adaptive weight for balancing the causal effects between new entity types and Other-Class. Experimental results on three benchmark datasets show that our method outperforms the state-of-the-art method by a large margin. Moreover, our method can be combined with the existing state-of-the-art methods to improve the performance in CL-NER.

Authors

Junhao Zheng

2 papers

Zhanxian Liang

1 papers

Haibin Chen

1 papers

TL;DR

Abstract

Authors

References43 items

CONTaiNER: Few-Shot Named Entity Recognition via Contrastive Learning

Uncovering Main Causalities for Long-tailed Information Extraction

De-biasing Distantly Supervised Named Entity Recognition via Causal Intervention

Continual Learning for Named Entity Recognition

Distilling Causal Effect of Data in Class-Incremental Learning

Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect

Causal Intervention for Weakly-Supervised Semantic Segmentation

PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning

Counterfactual Off-Policy Training for Neural Dialogue Generation

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Causality for Machine Learning

Dice Loss for Data-imbalanced NLP Tasks

Continual learning: A comparative study on how to defy forgetting in classification tasks

Learning a Unified Classifier Incrementally via Rebalancing

Unsupervised label noise modeling and loss correction

CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images

The curse(s) of dimensionality

Continual Lifelong Learning with Neural Networks: A Review

Overcoming catastrophic forgetting in neural networks

iCaRL: Incremental Classifier and Representation Learning

Causal Inference in Statistics: A Primer

End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF

Distilling the Knowledge in a Neural Network

An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks

Interpretation and Identification of Causal Mediation

Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2)

Curriculum learning

OntoNotes: The 90% Solution

Semi-Supervised Self-Training of Object Detection Models

Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

Lifelong Learning Algorithms

Catastrophic Forgetting, Rehearsal and Pseudorehearsal

Causality

hyper-parameters in BERT: hidden dimensions are 768, and the max sequence length is 512

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Greedy Sampling Algorithm for CL-NER In real-world scenarios, the new data should focus on the new entity

representing that assigning a certain value to a variable without considering all parent nodes (causes)

Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem

blocked by the collider X 0 . From Eq.

differences in prediction Y when the old data S exists or not

P ( ˆ Y = ˆ y ) − P ( ˆ Y = ˆ y ) ( = 0 , ( where do ( · ) is the causal intervention

identify the cause of forgetting

P ( ˆ Y = ˆ y | S ) due to the fact that S has no parent nodes

Field of Study

Venue Information

Name

Type

URL

Alternate Names