FastText.zip: Compressing text classification models

Published in

arXiv.org(2016)

External Links:

Generate Graph

TL;DR

This work proposes a method built upon product quantization to store the word embeddings, which produces a text classifier, derived from the fastText approach, which at test time requires only a fraction of the memory compared to the original one, without noticeably sacrificing the quality in terms of classification accuracy.

Abstract

We consider the problem of producing compact architectures for text classification, such that the full model fits in a limited amount of memory. After considering different solutions inspired by the hashing literature, we propose a method built upon product quantization to store the word embeddings. While the original technique leads to a loss in accuracy, we adapt this method to circumvent the quantization artifacts. As a result, our approach produces a text classifier, derived from the fastText approach, which at test time requires only a fraction of the memory compared to the original one, without noticeably sacrificing the quality in terms of classification accuracy. Our experiments carried out on several benchmarks show that our approach typically requires two orders of magnitude less memory than fastText while being only slightly inferior with respect to accuracy. As a result, it outperforms the state of the art by a good margin in terms of the compromise between memory usage and accuracy.

Authors

Edouard Grave

12 papers

Piotr Bojanowski

15 papers

Armand Joulin

18 papers

References45 items

How should we evaluate supervised hashing?

Efficient softmax approximation for GPUs

Bag of Tricks for Efficient Text Classification

Binarized Neural Networks

Efficient Character-level Document Classification by Combining Convolution and Recurrent Layers

FastText.zip: Compressing text classification models

Published in

arXiv.org(2016)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Edouard Grave

12 papers

Piotr Bojanowski

15 papers

Armand Joulin

18 papers

References45 items

How should we evaluate supervised hashing?

Efficient softmax approximation for GPUs

Bag of Tricks for Efficient Text Classification

Binarized Neural Networks

Efficient Character-level Document Classification by Combining Convolution and Recurrent Layers

Matthijs Douze

10 papers

Tomas Mikolov

9 papers

Strategies for Training Large Vocabulary Neural Language Models

Neural Networks with Few Multiplications

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Learning to Hash for Indexing Big Data—A Survey

Character-level Convolutional Networks for Text Classification

Compressing Deep Convolutional Networks using Vector Quantization

On Symmetric and Asymmetric LSHs for Inner Product Search

Streaming submodular maximization: massive data summarization on the fly

Hashing for Similarity Search: A Survey

Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)

Optimized Product Quantization for Approximate Nearest Neighbor Search

Cartesian K-Means

Predicting Parameters in Deep Learning

A new approach to interdomain routing based on secure multi-party computation

Baselines and Bigrams: Simple, Good Sentiment and Topic Classification

Optimization with Sparsity-Inducing Penalties (Foundations and Trends(R) in Machine Learning)

A reliable effective terascale linear learning system

Optimization with Sparsity-Inducing Penalties

High-dimensional signature compression for large-scale image classification

Iterative quantization: A procrustean approach to learning binary codes

Submodular secretary problem and extensions

Max-cover in map-reduce

Feature hashing for large scale multitask learning

Latent semantic analysis

Hamming Embedding and Weak Geometric Consistency for Large Scale Image Search

Randomized Language Models via Perfect Hash Functions

The group lasso for logistic regression

Locality-sensitive hashing scheme based on p-stable distributions

Similarity estimation techniques from rounding algorithms

Entropy-based Pruning of Backoff Language Models

Text Categorization with Support Vector Machines: Learning with Many Relevant Features

A threshold of ln n for approximating set cover (preliminary version)

Opinion Mining and Sentiment Analysis

Yfcc100m: The new data in multimedia research

Statistical Language Models Based on Neural Networks

Product Quantization for Nearest Neighbor Search

SUBWORD LANGUAGE MODELING WITH NEURAL NETWORKS

A comparison of event models for naive bayes text classification

Optimal Brain Damage

Under review as a conference paper at ICLR 2019 softmax Transpose Updated Memory Memory embedded input linear concat Attention Weights concat

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

arXiv.org

Type

URL

https://arxiv.org

Alternate Names

ArXiv