ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Published in

International Conference on Learning Representa...(2019)

External Links:

Generate Graph

TL;DR

This work presents two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT, and uses a self-supervised loss that focuses on modeling inter-sentence coherence.

Abstract

Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and \squad benchmarks while having fewer parameters compared to BERT-large. The code and the pretrained models are available at this https URL.

Authors

Kevin Gimpel

12 papers

Zhenzhong Lan

2 papers

Radu Soricut

17 papers

References72 items

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Deep Equilibrium Models

Patient Knowledge Distillation for BERT Model Compression

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Published in

International Conference on Learning Representa...(2019)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Kevin Gimpel

12 papers

Zhenzhong Lan

2 papers

Radu Soricut

17 papers

References72 items

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Deep Equilibrium Models

Patient Knowledge Distillation for BERT Model Compression

Piyush Sharma

3 papers

Sebastian Goodman

5 papers

Mingda Chen

6 papers

Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation

StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding

RoBERTa: A Robustly Optimized BERT Pretraining Approach

SpanBERT: Improving Pre-training by Representing and Predicting Spans

BAM! Born-Again Multi-Task Networks for Natural Language Understanding

DisSent: Learning Sentence Representations from Explicit Discourse Relations

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Efficient Training of BERT by Progressively Stacking

Generating Long Sequences with Sparse Transformers

Modeling Recurrence for Transformer

Reducing BERT Pre-Training Time from 3 Days to 76 Minutes

Dual Co-Matching Network for Multi-choice Reading Comprehension

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Mesh-TensorFlow: Deep Learning for Supercomputers

Adaptive Input Representations for Neural Language Modeling

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Know What You Don’t Know: Unanswerable Questions for SQuAD

A Simple Method for Commonsense Reasoning

Neural Network Acceptability Judgments

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling

Deep Contextualized Word Representations

Universal Language Model Fine-tuning for Text Classification

Understanding the Disharmony Between Dropout and Batch Normalization by Variance Shift

Learned in Translation: Contextualized Word Vectors

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

The Reversible Residual Network: Backpropagation Without Storing Activations

Attention is All you Need

Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

RACE: Large-scale ReAding Comprehension Dataset From Examinations

Learning Generic Sentence Representations Using Convolutional Neural Networks

Efficient softmax approximation for GPUs

Gaussian Error Linear Units (GELUs)

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Training Deep Nets with Sublinear Memory Cost

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Learning Distributed Representations of Sentences from Unlabelled Data

Semi-supervised Sequence Learning

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Skip-Thought Vectors

GloVe: Global Vectors for Word Representation

Distributed Representations of Sentences and Documents

Distributed Representations of Words and Phrases and their Compositionality

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

The Winograd Schema Challenge

Centering: A Framework for Modeling the Local Coherence of Discourse

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

2019), and Yang et al. (2019)

Improving Language Understanding by Generative Pre-Training

First quora dataset release: Ques

DOWNSTREAM EVALUATION TASKS GLUE GLUE is comprised of 9 tasks, namely Corpus of Linguistic Acceptability (CoLA

The Seventh PASCAL Recognizing Textual Entailment Challenge

The Sixth PASCAL Recognizing Textual Entailment Challenge

The Fourth PASCAL Recognizing Textual Entailment Challenge

The Second PASCAL Recognising Textual Entailment Challenge

Automatically Constructing a Corpus of Sentential Paraphrases

Semantic Textual Similarity Benchmark

The PASCAL Recognising Textual Entailment Challenge

Coherence and Coreference

Cohesion in English

the difference between a 12-layer and a 24-layer configuration is small. Does this result still hold for much wider ALBERT configurations, such as ALBERT-xxlarge (H=4096)? Number of layers

The difference between 12-layer and 24-layer ALBERT-xxlarge configurations in terms of downstream accuracy is negligible, with the Avg score being the same

A.2 DO VERY WIDE ALBERT MODELS NEED TO BE DEEP(ER) TOO?

of the Association for Computational Linguistics

The effect of a deeper network using an ALBERT-xxlarge configuration

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

International Conference on Learning Representations

Type

conference

URL

https://iclr.cc/

Alternate Names

Int Conf Learn Represent
ICLR