Refining Sample Embeddings with Relation Prototypes to Enhance Continual Relation Extraction (2021-01-01T00:00:00.000000Z)

TL;DR

The proposed model outperforms the state-of-the-art CRE models and has great advantage in avoiding catastrophic forgetting, as well as utilizing the memory information sufficiently and efficiently, resulting in enhanced CRE performance.

Abstract

Continual learning has gained increasing attention in recent years, thanks to its biological interpretation and efficiency in many real-world applications. As a typical task of continual learning, continual relation extraction (CRE) aims to extract relations between entities from texts, where the samples of different relations are delivered into the model continuously. Some previous works have proved that storing typical samples of old relations in memory can help the model keep a stable understanding of old relations and avoid forgetting them. However, most methods heavily depend on the memory size in that they simply replay these memorized samples in subsequent tasks. To fully utilize memorized samples, in this paper, we employ relation prototype to extract useful information of each relation. Specifically, the prototype embedding for a specific relation is computed based on memorized samples of this relation, which is collected by K-means algorithm. The prototypes of all observed relations at current learning stage are used to re-initialize a memory network to refine subsequent sample embeddings, which ensures the model’s stable understanding on all observed relations when learning a new task. Compared with previous CRE models, our model utilizes the memory information sufficiently and efficiently, resulting in enhanced CRE performance. Our experiments show that the proposed model outperforms the state-of-the-art CRE models and has great advantage in avoiding catastrophic forgetting. The code and datasets are released on https://github.com/fd2014cl/RP-CRE.

Authors

References46 items

Attention-based bidirectional long short-term memory networks for extracting temporal relationships from clinical discharge summaries

Prototypical Representation Learning for Relation Extraction

Curriculum-Meta Learning for Order-Robust Continual Relation Extraction

Embracing Change: Continual Learning in Deep Neural Networks

Transformers: State-of-the-Art Natural Language Processing

Continual Relation Learning via Episodic Memory Activation and Reconsolidation

A Novel Cascade Binary Tagging Framework for Relational Triple Extraction

Meta-Learning Improves Lifelong Relation Extraction

Hybrid Attention-Based Prototypical Networks for Noisy Few-Shot Relation Classification

GraphRel: Modeling Text as Relational Graphs for Joint Entity and Relation Extraction

Matching the Blanks: Distributional Similarity for Relation Learning

Enriching Pre-trained Language Model with Entity Information for Relation Classification

Sentence Embedding Alignment for Lifelong Relation Extraction

Graph Neural Networks with Generated Parameters for Relation Extraction

FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation

Efficient Lifelong Learning with A-GEM

Graph Convolution over Pruned Dependency Trees Improves Relation Extraction

Relational recurrent neural networks

Relation classification via recurrent neural network with attention and tensor layers

Continual Lifelong Learning with Neural Networks: A Review

Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting

Memory Aware Synapses: Learning what (not) to forget

Position-aware Attention and Supervised Data Improve Slot Filling

Neuroscience-Inspired Artificial Intelligence

Gradient Episodic Memory for Continual Learning

Prototypical Networks for Few-shot Learning

Continual Learning Through Synaptic Intelligence

Distant Supervision for Relation Extraction with Sentence-Level Attention and Entity Descriptions

PathNet: Evolution Channels Gradient Descent in Super Neural Networks

Overcoming catastrophic forgetting in neural networks

Hybrid computing using a neural network with dynamic external memory

Neural Relation Extraction with Selective Attention over Instances

Meta-Learning with Memory-Augmented Neural Networks

Net2Net: Accelerating Learning via Knowledge Transfer

Distant Supervision for Relation Extraction via Piecewise Convolutional Neural Networks

Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Paths

Relation Classification via Recurrent Neural Network

Relation Extraction: Perspective from Convolutional Neural Networks

Relation Classification via Convolutional Deep Neural Network

Catastrophic forgetting in connectionist networks

Long Short-Term Memory

Lifelong robot learning

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars

Relation Extraction with Matrix Factorization and Universal Schemas

Visualizing Data using t-SNE