Knowledge Inheritance for Pre-trained Language Models (2021-05-28T00:00:00.000000Z)

TL;DR

A pre-training framework named “knowledge inheritance” (KI) is introduced and how could knowledge distillation serve as auxiliary supervision during pre- training to efficiently learn larger PLMs is explored, demonstrating the superiority of KI in training efficiency.

Abstract

Recent explorations of large-scale pre-trained language models (PLMs) have revealed the power of PLMs with huge amounts of parameters, setting off a wave of training ever-larger PLMs. However, it requires tremendous computational resources to train a large-scale PLM, which may be practically unaffordable. In addition, existing large-scale PLMs are mainly trained from scratch individually, ignoring that many well-trained PLMs are available. To this end, we explore the question how could existing PLMs benefit training large-scale PLMs in future. Specifically, we introduce a pre-training framework named “knowledge inheritance” (KI) and explore how could knowledge distillation serve as auxiliary supervision during pre-training to efficiently learn larger PLMs. Experimental results demonstrate the superiority of KI in training efficiency. We also conduct empirical analyses to explore the effects of teacher PLMs’ pre-training settings, including model architecture, pre-training data, etc. Finally, we show that KI could be applied to domain adaptation and knowledge transfer.

Authors

Jiajie Zhang

3 papers

Xu Han

16 papers

Zhiyuan Liu

34 papers

TL;DR

Abstract

Authors

References70 items

ELLE: Efficient Lifelong Pre-training for Emerging Data

Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey

bert2BERT: Towards Reusable Pretrained Language Models

On the Opportunities and Risks of Foundation Models

Pre-Trained Models: Past, Present and Future

A Survey of Transformers

Are Pretrained Convolutions Better than Pretrained Transformers?

PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

Carbon Emissions and Large Neural Network Training

Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning

Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping

On the Transformer Growth for Progressive BERT Training

Contrastive Distillation on Intermediate Representations for Language Model Compression

TernaryBERT: Distillation-aware Ultra-low Bit BERT

Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers

Revisiting Knowledge Distillation via Label Smoothing Regularization

Language Models are Few-Shot Learners

Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

Pre-trained models for natural language processing: A survey

Scaling Laws for Neural Language Models

KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

Distilling Knowledge Learned in BERT for Text Generation

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Thieves on Sesame Street! Model Extraction of BERT-based APIs

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

TinyBERT: Distilling BERT for Natural Language Understanding

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Knowledge Enhanced Contextual Word Representations

Patient Knowledge Distillation for BERT Model Compression

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Green AI

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Efficient Training of BERT by Progressively Stacking

ERNIE: Enhanced Language Representation with Informative Entities

Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints

MASS: Masked Sequence to Sequence Pre-training for Language Generation

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

Reducing BERT Pre-Training Time from 3 Days to 76 Minutes

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Multilingual Neural Machine Translation with Knowledge Distillation

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

Mesh-TensorFlow: Deep Learning for Supercomputers

Measuring the Evolution of a Scientific Field through Citation Frames

Training Deep Neural Networks in Generations: A More Tolerant Teacher Educates Better Students

Born Again Neural Networks

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Deep Contextualized Word Representations

Mixed Precision Training

Learned in Translation: Contextualized Word Vectors

Attention is All you Need

ChemProt-3.0: a global chemical biology diseases mapping

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Distilling the Knowledge in a Neural Network

Adam: A Method for Stochastic Optimization

GloVe: Global Vectors for Word Representation

Efficient Estimation of Word Representations in Vector Space

Exploring the Limits

The transformer.

S2ORC: The Semantic Scholar Open Research Corpus

8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

2019. BERT: Pre-training

Improving Language Understanding by Generative Pre-Training

Training Deep Neural Networks

Electra : Pre - training text encoders as discriminators rather than generators

Field of Study

Venue Information

Name

Type

URL

Alternate Names