Flexible Dataset Distillation: Learn Labels Instead of Images (2020-06-15T00:00:00.000000Z)

TL;DR

This work introduces a more robust and flexible meta-learning algorithm for distillation, as well as an effective first-order strategy based on convex optimization layers, and shows it to be more effective than the prior image-based approach to dataset distillation.

Abstract

We study the problem of dataset distillation - creating a small set of synthetic examples capable of training a good model. In particular, we study the problem of label distillation - creating synthetic labels for a small set of real images, and show it to be more effective than the prior image-based approach to dataset distillation. Interestingly, label distillation can be applied across datasets, for example enabling learning Japanese character recognition by training only on synthetically labeled English letters. Methodologically, we introduce a more robust and flexible meta-learning algorithm for distillation, as well as an effective first-order strategy based on convex optimization layers. Distilling labels with our new algorithm leads to improved results over prior image-based distillation. More importantly, it leads to clear improvements in flexibility of the distilled dataset in terms of compatibility with off-the-shelf optimizers and diverse neural architectures.

Authors

Timothy M. Hospedales

14 papers

Yongxin Yang

10 papers

Ondrej Bohdal

2 papers

TL;DR

Abstract

Authors

References37 items

'Less Than One'-Shot Learning: Learning N Classes From M

Meta-Learning in Neural Networks: A Survey

Optimizing Millions of Hyperparameters by Implicit Differentiation

Soft-Label Dataset Distillation and Text Dataset Distillation

Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Estimators for Reinforcement Learning

Green AI

Taming MAML: Efficient unbiased meta-reinforcement learning

Zero-shot Knowledge Transfer via Adversarial Belief Matching

Meta-Learning With Differentiable Convex Optimization

Feature-Critic Networks for Heterogeneous Domain Generalization

Deep Learning for Classical Japanese Literature

MetaReg: Towards Domain Generalization using Meta-Regularization

Dataset Distillation

Meta-learning with differentiable closed-form solvers

Privacy-Preserving Machine Learning: Threats and Solutions

On First-Order Meta-Learning Algorithms

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

Active Learning for Convolutional Neural Networks: A Core-Set Approach

Practical Coreset Constructions for Machine Learning

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

EMNIST: an extension of MNIST to handwritten letters

Regularizing Neural Networks by Penalizing Confident Output Distributions

Rethinking the Inception Architecture for Computer Vision

Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters

Distilling the Knowledge in a Neural Network

Gradient-based Hyperparameter Optimization through Reversible Learning

Are all training examples equally valuable?

ImageNet classification with deep convolutional neural networks

The Caltech-UCSD Birds-200-2011 Dataset

Object Detection with Discriminatively Trained Part Based Models

A review of instance selection methods

Core Vector Machines: Fast SVM Training on Very Large Data Sets

Pruning training sets for learning of object categories

Neural architecture search: a survey

Learning Multiple Layers of Features from Tiny Images

The Matrix Cookbook

Gradient-based learning applied to document recognition

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names