Soft-Label Dataset Distillation and Text Dataset Distillation (2019-10-06T00:00:00.000000Z)

TL;DR

This work proposes to simultaneously distill both images and their labels, thus assigning each synthetic sample a `soft' label (a distribution of labels) and demonstrates that text distillation outperforms other methods across multiple datasets.

Abstract

Dataset distillation is a method for reducing dataset sizes by learning a small number of representative synthetic samples. This has several benefits such as speeding up model training, reducing energy consumption, and reducing required storage space. These benefits are especially crucial in settings like federated learning where initial overhead costs are justified by the speedup they enable. Currently, 1) each synthetic sample is assigned a single ‘hard’ label, and 2) dataset distillation can only be used with image data. We propose to simultaneously distill both images and their labels, thus assigning each synthetic sample a ‘soft’ label (a distribution of labels). Our algorithm increases accuracy by 2-4% for several image classification tasks. Using ‘soft’ labels also enables distilled datasets to consist of fewer samples than there are classes as each sample encodes information for multiple classes. For example, training a LeNet model with 10 distilled images (one per class) results in over 96% accuracy on MNIST, and almost 92% accuracy when trained on just 5 distilled images. We also extend the dataset distillation algorithm to distill text data. We demonstrate that text distillation outperforms other methods across multiple datasets. For example, models attain almost their original accuracy on the IMDB sentiment analysis task using just 20 distilled sentences. Our code can be found at https://github.com/ilia10000/dataset-distillation.

Authors

Matthias Schonlau

2 papers

Ilia Sucholutsky

1 papers

TL;DR

Abstract

Authors

References54 items

Soft-Label Anonymous Gastric X-Ray Image Distillation

Loosely Coupled Federated Learning Over Generative Models

SecDD: Efficient and Secure Method for Remotely Training Neural Networks

Distilled One-Shot Federated Learning

Federated Learning via Synthetic Data

Federated Mutual Learning

Federated Learning With Quantized Global Model Updates

Language Models are Few-Shot Learners

Advances and Open Problems in Federated Learning

Federated Learning: Challenges, Methods, and Future Directions

Energy and Policy Considerations for Deep Learning in NLP

Dataset Distillation

Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion

Data-Efficient Hierarchical Reinforcement Learning

Measuring the Intrinsic Dimension of Objective Landscapes

Deep Contextualized Word Representations

StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation

Active Learning for Convolutional Neural Networks: A Core-Set Approach

Practical Coreset Constructions for Machine Learning

Prototypical Networks for Few-shot Learning

Optimization as a Model for Few-Shot Learning

SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Sequence-Level Knowledge Distillation

Safe and Efficient Off-Policy Reinforcement Learning

Generative Adversarial Text to Image Synthesis

End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF

Human-level concept learning through probabilistic program induction

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Distilling the Knowledge in a Neural Network

GloVe: Global Vectors for Word Representation

Generative Adversarial Nets

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Regularization of Neural Networks using DropConnect

Prototype Selection for Nearest Neighbor Classification: Taxonomy and Empirical Study

Learning Word Vectors for Sentiment Analysis

Understanding the difficulty of training deep feedforward neural networks

A Study of the Robustness of KNN Classifiers Trained Using Soft Labels

Model compression

Core Vector Machines: Fast SVM Training on Very Large Data Sets

Pruning training sets for learning of object categories

Support Vector Machine Active Learning with Applications to Text Classification

Long Short-Term Memory

Bidirectional recurrent neural networks

Active Learning with Statistical Models

Backpropagation Through Time: What It Does and How to Do It

THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS

Gradient-Based Learning

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

A Taxonomy and Experimental Study on Prototype Generation for Nearest Neighbor Classification

Learning Multiple Layers of Features from Tiny Images

Object Recognition with Gradient-Based Learning

The TREC-8 Question Answering Track Report

Gradient-based learning applied to document recognition

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names