BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Published in

North American Chapter of the Association for C...(2018)

External Links:

Generate Graph

TL;DR

A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.

Abstract

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Authors

Jacob Devlin

8 papers

Kenton Lee

10 papers

Kristina Toutanova

7 papers

References63 items

Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering

U-Net: Machine Reading Comprehension with Unanswerable Questions

Semi-Supervised Sequence Modeling with Cross-View Training

Dissecting Contextual Word Embeddings: Architecture and Representation

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Published in

North American Chapter of the Association for C...(2018)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Jacob Devlin

8 papers

Kenton Lee

10 papers

Kristina Toutanova

7 papers

References63 items

Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering

U-Net: Machine Reading Comprehension with Unanswerable Questions

Semi-Supervised Sequence Modeling with Cross-View Training

Dissecting Contextual Word Embeddings: Architecture and Representation

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

Ming-Wei Chang

11 papers

Character-Level Language Modeling with Deeper Self-Attention

Contextual String Embeddings for Sequence Labeling

Neural Network Acceptability Judgments

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Deep Contextualized Word Representations

An efficient framework for learning sentence representations

QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension

MaskGAN: Better Text Generation via Filling in the ______

Universal Language Model Fine-tuning for Text Classification

Simple and Effective Multi-Paragraph Reading Comprehension

Learned in Translation: Contextualized Word Vectors

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

Attention is All you Need

Reinforced Mnemonic Reader for Machine Reading Comprehension

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Semi-supervised sequence tagging with bidirectional language models

Bidirectional Attention Flow for Machine Comprehension

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

context2vec: Learning Generic Context Embedding with Bidirectional LSTM

Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units

SQuAD: 100,000+ Questions for Machine Comprehension of Text

A Decomposable Attention Model for Natural Language Inference

Learning Distributed Representations of Sentences from Unlabelled Data

Semi-supervised Sequence Learning

A large annotated corpus for learning natural language inference

Skip-Thought Vectors

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

How transferable are features in deep neural networks?

GloVe: Global Vectors for Word Representation

Distributed Representations of Sentences and Documents

One billion word benchmark for measuring progress in statistical language modeling

Distributed Representations of Words and Phrases and their Compositionality

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

The Winograd Schema Challenge

Word Representations: A Simple and General Method for Semi-Supervised Learning

Natural Language Understanding

ImageNet: A large-scale hierarchical image database

A Scalable Hierarchical Distributed Language Model

A unified architecture for natural language processing: deep neural networks with multitask learning

Extracting and composing robust features with denoising autoencoders

Domain Adaptation with Structural Correspondence Learning

A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data

Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

Class-Based n-gram Models of Natural Language

“Cloze Procedure”: A New Tool for Measuring Readability

Improving language understanding with unsupervised learning

Corpus of linguistic acceptability

Quora Question Pairs

2016) rather than the standard relu, following OpenAI GPT. The training loss is the sum of the mean masked LM likelihood and the mean next sentence prediction likelihood

TL;DR

Abstract

Authors

References63 items

Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering

U-Net: Machine Reading Comprehension with Unanswerable Questions

Semi-Supervised Sequence Modeling with Cross-View Training

Dissecting Contextual Word Embeddings: Architecture and Representation

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

TL;DR

Abstract

Authors

References63 items

Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering

U-Net: Machine Reading Comprehension with Unanswerable Questions

Semi-Supervised Sequence Modeling with Cross-View Training

Dissecting Contextual Word Embeddings: Architecture and Representation

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

Character-Level Language Modeling with Deeper Self-Attention

Contextual String Embeddings for Sequence Labeling

Neural Network Acceptability Judgments

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Deep Contextualized Word Representations

An efficient framework for learning sentence representations

QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension

MaskGAN: Better Text Generation via Filling in the ______

Universal Language Model Fine-tuning for Text Classification

Simple and Effective Multi-Paragraph Reading Comprehension

Learned in Translation: Contextualized Word Vectors

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

Attention is All you Need

Reinforced Mnemonic Reader for Machine Reading Comprehension

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Semi-supervised sequence tagging with bidirectional language models

Bidirectional Attention Flow for Machine Comprehension

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

context2vec: Learning Generic Context Embedding with Bidirectional LSTM

Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units

SQuAD: 100,000+ Questions for Machine Comprehension of Text

A Decomposable Attention Model for Natural Language Inference

Learning Distributed Representations of Sentences from Unlabelled Data

Semi-supervised Sequence Learning

A large annotated corpus for learning natural language inference

Skip-Thought Vectors

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

How transferable are features in deep neural networks?

GloVe: Global Vectors for Word Representation

Distributed Representations of Sentences and Documents

One billion word benchmark for measuring progress in statistical language modeling

Distributed Representations of Words and Phrases and their Compositionality

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

The Winograd Schema Challenge

Word Representations: A Simple and General Method for Semi-Supervised Learning

Natural Language Understanding

ImageNet: A large-scale hierarchical image database

A Scalable Hierarchical Distributed Language Model

A unified architecture for natural language processing: deep neural networks with multitask learning

Extracting and composing robust features with denoising autoencoders

Domain Adaptation with Structural Correspondence Learning

A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data

Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

Class-Based n-gram Models of Natural Language

“Cloze Procedure”: A New Tool for Measuring Readability

Improving language understanding with unsupervised learning

Corpus of linguistic acceptability

Quora Question Pairs

2016) rather than the standard relu, following OpenAI GPT. The training loss is the sum of the mean masked LM likelihood and the mean next sentence prediction likelihood

The Seventh PASCAL Recognizing Textual Entailment Challenge

The Sixth PASCAL Recognizing Textual Entailment Challenge

The Fourth PASCAL Recognizing Textual Entailment Challenge

Automatically Constructing a Corpus of Sentential Paraphrases

The PASCAL Recognising Textual Entailment Challenge

We provide examples of the pre-training tasks in the following

Field of Study

Venue Information

Name

Type