ERNIE: Enhanced Language Representation with Informative Entities (2019-05-17T00:00:00.000000Z)

TL;DR

This paper utilizes both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE) which can take full advantage of lexical, syntactic, and knowledge information simultaneously, and is comparable with the state-of-the-art model BERT on other common NLP tasks.

Abstract

Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding. We argue that informative entities in KGs can enhance language representation with external knowledge. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. The experimental results have demonstrated that ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks. The code and datasets will be available in the future.

Authors

Xu Han

16 papers

Zhiyuan Liu

34 papers

Maosong Sun

46 papers

TL;DR

Abstract

Authors

References59 items

ERNIE: Enhanced Representation through Knowledge Integration

Joint Representation Learning of Cross-lingual Words and Entities via Attentive Distant Supervision

FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation

Graph Convolution over Pruned Dependency Trees Improves Relation Extraction

Improving Question Answering by Commonsense-Based Pre-Training

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

Ultra-Fine Entity Typing

Adaptive Knowledge Sharing in Multi-Task Learning: Improving Low-Resource Neural Machine Translation

Neural Network Acceptability Judgments

Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge

Neural Knowledge Acquisition via Mutual Attention Between Knowledge Graph and Text

Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Deep Contextualized Word Representations

Universal Language Model Fine-tuning for Text Classification

Neural Natural Language Inference Models Enhanced with External Knowledge

Position-aware Attention and Supervised Data Improve Slot Filling

Adversarial Training for Relation Extraction

Regularizing and Optimizing LSTM Language Models

Attention is All you Need

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Semi-supervised sequence tagging with bidirectional language models

Multi-level Representations for Fine-Grained Typing of Knowledge Base Entities

Joint Representation Learning of Text and Knowledge for Knowledge Graph Completion

Neural Relation Extraction with Selective Attention over Instances

Gaussian Error Linear Units (GELUs)

SQuAD: 100,000+ Questions for Machine Comprehension of Text

An Attentive Neural Architecture for Fine-grained Entity Type Classification

Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation

Semi-supervised Sequence Learning

Distant Supervision for Relation Extraction via Piecewise Convolutional Neural Networks

A large annotated corpus for learning natural language inference

Design Challenges for Entity Linking

GloVe: Global Vectors for Word Representation

Knowledge Graph and Text Jointly Embedding

Translating Embeddings for Modeling Multi-relational Data

Distributed Representations of Words and Phrases and their Compositionality

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

The Winograd Schema Challenge

Representing General Relational Knowledge in ConceptNet 5

Word Representations: A Simple and General Method for Semi-Supervised Learning

TAGME: on-the-fly annotation of short text fragments (by wikipedia entities)

A unified architecture for natural language processing: deep neural networks with multitask learning

Extracting and composing robust features with denoising autoencoders

The Third PASCAL Recognizing Textual Entailment Challenge

Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Hierarchical Relation Extraction with Coarse-to-Fine Grained Attention

Improving Language Understanding by Generative Pre-Training

Put It Back: Entity Typing with Language Model Enhancement

Bridge Text and Knowledge by Learning Multi-Prototype Entity Mention Embedding

Representing Text for Joint Embedding of Text and Knowledge Bases

The Seventh PASCAL Recognizing Textual Entailment Challenge

The Sixth PASCAL Recognizing Textual Entailment Challenge

Proceedings of the fourth international workshop on semantic evaluations (semeval2007)

Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment

The Second PASCAL Recognising Textual Entailment Challenge

The PASCAL Recognising Textual Entailment Challenge

Automatically Constructing a Corpus of Sentential Paraphrases

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names