CPM: A Large-scale Generative Chinese Pre-trained Language Model (2020-12-01T00:00:00.000000Z)

TL;DR

CPM, with 2.6 billion parameters and 100GB Chinese training data, is the largest Chinese pre-trained language model, which could facilitate several downstream Chinese NLP tasks, such as conversation, essay generation, cloze test, and language understanding.

Authors

Jie Tang

8 papers

Xu Han

16 papers

Minlie Huang

23 papers

TL;DR

Authors

References42 items

Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads

OCNLI: Original Chinese Natural Language Inference

Contextual Knowledge Selection and Embedding towards Enhanced Pre-Trained Language Models

A Large-Scale Chinese Short-Text Conversation Dataset

DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters

On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines

Language Models are Few-Shot Learners

CLUE: A Chinese Language Understanding Evaluation Benchmark

Revisiting Pre-Trained Models for Chinese Natural Language Processing

Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping

Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model

KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models

TinyBERT: Distilling BERT for Natural Language Understanding

Knowledge Enhanced Contextual Word Representations

NEZHA: Neural Contextualized Representation for Chinese Language Understanding

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Pre-Training with Whole Word Masking for Chinese BERT

ChID: A Large-scale Chinese IDiom Dataset for Cloze Test

ERNIE: Enhanced Language Representation with Informative Entities

The Curious Case of Neural Text Degeneration

ERNIE: Enhanced Representation through Knowledge Integration

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Generating Informative Responses with Controlled Sentence Function

Deep Contextualized Word Representations

DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications

Attention is All you Need

Topic Aware Neural Response Generation

SQuAD: 100,000+ Questions for Machine Comprehension of Text

How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

A Diversity-Promoting Objective Function for Neural Conversation Models

Neural Responding Machine for Short-Text Conversation

Adam: A Method for Stochastic Optimization

SentiLARE: sentiment-aware language representation learning with linguistic knowledge

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

A Span-Extraction Dataset for Chinese Machine Reading Comprehension

Language Models are Unsupervised Multitask Learners

Improving Language Understanding by Generative Pre-Training

collected, ﬁltered, deduplicated training data

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL