Universal Language Model Fine-tuning for Text Classification

Published in

Annual Meeting of the Association for Computati...(2018)

External Links:

Generate Graph DownloadPDF

TL;DR

This work proposes Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduces techniques that are key for fine- Tuning a language model.

Abstract

Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a language model. Our method significantly outperforms the state-of-the-art on six text classification tasks, reducing the error by 18-24% on the majority of datasets. Furthermore, with only 100 labeled examples, it matches the performance of training from scratch on 100 times more data. We open-source our pretrained models and code.

Authors

Sebastian Ruder

20 papers

Jeremy Howard

3 papers

References57 items

Exploring the Limits of Weakly Supervised Pretraining

Colorless Green Recurrent Networks Dream Hierarchically

Deep Contextualized Word Representations

GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks

Empower Sequence Labeling with Task-Aware Neural Language Model

Universal Language Model Fine-tuning for Text Classification

Published in

Annual Meeting of the Association for Computati...(2018)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Sebastian Ruder

20 papers

Jeremy Howard

3 papers

References57 items

Exploring the Limits of Weakly Supervised Pretraining

Colorless Green Recurrent Networks Dream Hierarchically

Deep Contextualized Word Representations

GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks

Empower Sequence Labeling with Task-Aware Neural Language Model

Super-Convergence: Very Fast Training of Residual Networks Using Large Learning Rates

Regularizing and Optimizing LSTM Language Models

Learned in Translation: Contextualized Word Vectors

Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm

Deep Pyramid Convolutional Neural Networks for Text Categorization

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings

Semi-supervised Multitask Learning for Sequence Labeling

Learning to Generate Reviews and Discovering Sentiment

Semi-supervised sequence tagging with bidirectional language models

Question Answering through Transfer Learning from Large Fine-grained Supervision Data

Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling

Deep Biaffine Attention for Neural Dependency Parsing

Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

Pointer Sentinel Mixture Models

An overview of gradient descent optimization algorithms

What makes ImageNet good for transfer learning?

Densely Connected Convolutional Networks

SGDR: Stochastic Gradient Descent with Warm Restarts

Adversarial Training Methods for Semi-Supervised Text Classification

How Transferable are Neural Networks in NLP Applications?

Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings

Deep Residual Learning for Image Recognition

Improving Neural Machine Translation Models with Monolingual Data

Semi-supervised Sequence Learning

Character-level Convolutional Networks for Text Classification

Cyclical Learning Rates for Training Neural Networks

UNITN: Training Deep Convolutional Neural Network for Twitter Sentiment Classification

Discriminative Neural Sentence Modeling by Tree-Based Convolution

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Learning Transferable Features with Deep Adaptation Networks

Hypercolumns for object segmentation and fine-grained localization

Fully convolutional networks for semantic segmentation

How transferable are features in deep neural networks?

CNN Features Off-the-Shelf: An Astounding Baseline for Recognition

Distributed Representations of Words and Phrases and their Compositionality

DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition

Detecting Automation of Twitter Accounts: Are You a Human, Bot, or Cyborg?

Learning Word Vectors for Sentiment Analysis

The application of data mining techniques in financial fraud detection: A classification framework and an academic review of literature

A Survey on Transfer Learning

Why Does Unsupervised Pre-training Help Deep Learning?

Deep Boltzmann Machines

Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification

Review spam detection

Greedy Layer-Wise Training of Deep Networks

Estimation of Dependences Based on Empirical Data

The TREC-8 Question Answering Track Evaluation

A Model of Inductive Bias Learning

Multitask Learning: A Knowledge-Based Source of Inductive Bias

Classifying text messages for the haiti earthquake

Document categorization in legal electronic discovery: computer classification vs. manual review

Field of Study

Computer Science

Venue Information

Name

Annual Meeting of the Association for Computational Linguistics

Type

conference

URL

https://www.aclweb.org/anthology/venues/acl/

Alternate Names

Annu Meet Assoc Comput Linguistics
Meeting of the Association for Computational Linguistics
ACL
Meet Assoc Comput Linguistics