KEPLER: A Unifi… (2019-11-13T00:00:00.000000Z)

TL;DR

A unified model for Knowledge Embedding and Pre-trained LanguagERepresentation (KEPLER), which can not only better integrate factual knowledge into PLMs but also produce effective text-enhanced KE with the strong PLMs is proposed.

Abstract

Abstract Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text. In contrast, knowledge embedding (KE) methods can effectively represent the relational facts in knowledge graphs (KGs) with informative entity embeddings, but conventional KE models cannot take full advantage of the abundant textual information. In this paper, we propose a unified model for Knowledge Embedding and Pre-trained LanguagERepresentation (KEPLER), which can not only better integrate factual knowledge into PLMs but also produce effective text-enhanced KE with the strong PLMs. In KEPLER, we encode textual entity descriptions with a PLM as their embeddings, and then jointly optimize the KE and language modeling objectives. Experimental results show that KEPLER achieves state-of-the-art performances on various NLP tasks, and also works remarkably well as an inductive KE model on KG link prediction. Furthermore, for pre-training and evaluating KEPLER, we construct Wikidata5M1 , a large-scale KG dataset with aligned entity descriptions, and benchmark state-of-the-art KE methods on it. It shall serve as a new KE benchmark and facilitate the research on large KG, inductive KE, and KG with text. The source code can be obtained from https://github.com/THU-KEG/KEPLER.

Authors

Zhiyuan Liu

34 papers

Juan-Zi Li

8 papers

Tianyu Gao

7 papers

References76 items

K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model

Zero-shot Entity Linking with Dense Entity Retrieval

E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT

Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly

Abstract

References76 items

TL;DR

Abstract

Authors

References76 items

K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model

Zero-shot Entity Linking with Dense Entity Retrieval

E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT

Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly

TL;DR

Abstract

Authors

References76 items

K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model

Zero-shot Entity Linking with Dense Entity Retrieval

E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT

Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

FewRel 2.0: Towards More Challenging Few-Shot Relation Classification

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

K-BERT: Enabling Language Representation with Knowledge Graph

Knowledge Enhanced Contextual Word Representations

Language Models as Knowledge Bases?

Latent Relation Language Models

RoBERTa: A Robustly Optimized BERT Pretraining Approach

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zero-Shot Entity Linking by Reading Entity Descriptions

Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling

Matching the Blanks: Distributional Similarity for Relation Learning

ERNIE: Enhanced Language Representation with Informative Entities

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

GraphVite: A High-Performance CPU-GPU Hybrid System for Node Embedding

Linguistic Knowledge and Transferability of Contextual Representations

Multi-Task Deep Neural Networks for Natural Language Understanding

TuckER: Tensor Factorization for Knowledge Graph Completion

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling

Joint Representation Learning of Cross-lingual Words and Entities via Attentive Distant Supervision

Logic Attention Based Neighborhood Aggregation for Inductive Knowledge Graph Embedding

FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation

RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space

Improving Question Answering by Commonsense-Based Pre-Training

Adaptive Knowledge Sharing in Multi-Task Learning: Improving Low-Resource Neural Machine Translation

Ultra-Fine Entity Typing

Accurate Text-Enhanced Knowledge Graph Representation Learning

Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge

Neural Knowledge Acquisition via Mutual Attention Between Knowledge Graph and Text

Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Deep Contextualized Word Representations

SimplE Embedding for Link Prediction in Knowledge Graphs

Universal Language Model Fine-tuning for Text Classification

Open-World Knowledge Graph Completion

Position-aware Attention and Supervised Data Improve Slot Filling

Regularizing and Optimizing LSTM Language Models

Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking

Leveraging Knowledge Bases in LSTMs for Improving Machine Reading

Knowledge Transfer for Out-of-Knowledge-Base Entities : A Graph Neural Network Approach

Attention is All you Need

Inductive Representation Learning on Large Graphs

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Semi-supervised sequence tagging with bidirectional language models

Prototypical Networks for Few-shot Learning

Joint Representation Learning of Text and Knowledge for Knowledge Graph Completion

Complex Embeddings for Simple Link Prediction

Representation Learning of Knowledge Graphs with Entity Descriptions

Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation

Semi-supervised Sequence Learning

Neural Machine Translation of Rare Words with Subword Units

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Learning Entity and Relation Embeddings for Knowledge Graph Completion

Embedding Entities and Relations for Learning and Inference in Knowledge Bases

Knowledge Graph and Text Jointly Embedding

GloVe: Global Vectors for Word Representation

Translating Embeddings for Modeling Multi-relational Data

Distributed Representations of Words and Phrases and their Compositionality

A unified architecture for natural language processing: deep neural networks with multitask learning

WordNet: A Lexical Database for English

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding