LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention (2020-10-02T00:00:00.000000Z)

TL;DR

New pretrained contextualized representations of words and entities based on the bidirectional transformer, and an entity-aware self-attention mechanism that considers the types of tokens (words or entities) when computing attention scores are proposed.

Abstract

Entity representations are useful in natural language tasks involving entities. In this paper, we propose new pretrained contextualized representations of words and entities based on the bidirectional transformer. The proposed model treats words and entities in a given text as independent tokens, and outputs contextualized representations of them. Our model is trained using a new pretraining task based on the masked language model of BERT. The task involves predicting randomly masked words and entities in a large entity-annotated corpus retrieved from Wikipedia. We also propose an entity-aware self-attention mechanism that is an extension of the self-attention mechanism of the transformer, and considers the types of tokens (words or entities) when computing attention scores. The proposed model achieves impressive empirical performance on a wide range of entity-related tasks. In particular, it obtains state-of-the-art results on five well-known datasets: Open Entity (entity typing), TACRED (relation classification), CoNLL-2003 (named entity recognition), ReCoRD (cloze-style question answering), and SQuAD 1.1 (extractive question answering). Our source code and pretrained representations are available at this https URL.

Authors

Akari Asai

7 papers

Yuji Matsumoto

2 papers

Hideaki Takeda

2 papers

TL;DR

Abstract

Authors

References46 items

K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Learning Cross-Context Entity Representations from Text

Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model

KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Knowledge Enhanced Contextual Word Representations

RoBERTa: A Robustly Optimized BERT Pretraining Approach

SpanBERT: Improving Pre-training by Representing and Predicting Spans

XLNet: Generalized Autoregressive Pretraining for Language Understanding

What Does BERT Look at? An Analysis of BERT’s Attention

Matching the Blanks: Distributional Similarity for Relation Learning

Visualizing and Measuring the Geometry of BERT

ERNIE: Enhanced Language Representation with Informative Entities

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

Cloze-driven Pretraining of Self-attention Networks

ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension

Graph Convolution over Pruned Dependency Trees Improves Relation Extraction

Contextual String Embeddings for Sequence Labeling

Ultra-Fine Entity Typing

Deep Contextualized Word Representations

Simple and Effective Multi-Paragraph Reading Comprehension

Position-aware Attention and Supervised Data Improve Slot Filling

Attention is All you Need

Learning Distributed Representations of Texts and Entities from Knowledge Base

Deep Joint Entity Disambiguation with Local Neural Attention

Bidirectional Attention Flow for Machine Comprehension

Layer Normalization

Gaussian Error Linear Units (GELUs)

Complex Embeddings for Simple Link Prediction

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Neural Architectures for Named Entity Recognition

Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation

Embedding Entities and Relations for Learning and Inference in Knowledge Bases

Translating Embeddings for Modeling Multi-relational Data

Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

We optimize the model using AdamW with learning rate warmup and linear decay of the learning

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Pingan Smart Health and SJTU at COIN - Shared Task: utilizing Pre-trained Language Models and Common-sense Knowledge in Machine Reading Tasks

Commonsense Inference in Natural Language Processing (COIN) - Shared Task Report

The details of the datasets used in our experiments are provided below. B.1 Open Entity The Open Entity dataset

TACRED The TACRED dataset contains 68,124 training examples, 22,631 development examples, and 15,509 test examples with labels of their relation types

Deep Exhaustive Model for Nested Named Entity Recognition

Bridge Text and Knowledge by Learning Multi-Prototype Entity Mention Embedding

treat all matched entity names as entity annotations for their referent entities

Field of Study

Venue Information

Name

Type

URL

Alternate Names