Data Level Lottery Ticket Hypothesis for Vision Transformers (2022-11-02T00:00:00.000000Z)

TL;DR

This paper shows that the conventional winning ticket is hard to find at weight level of ViTs by existing methods, and generalizes the LTH for ViTs to input data consisting of image patches inspired by the input dependence ofViTs, which verifies the integrity of the theory.

Abstract

The conventional lottery ticket hypothesis (LTH) claims that there exists a sparse subnetwork within a dense neural network and a proper random initialization method, called the winning ticket, such that it can be trained from scratch to almost as good as the dense counterpart. Meanwhile, the research of LTH in vision transformers (ViTs) is scarcely evaluated. In this paper, we first show that the conventional winning ticket is hard to find at weight level of ViTs by existing methods. Then, we generalize the LTH for ViTs to input data consisting of image patches inspired by the input dependence of ViTs. That is, there exists a subset of input image patches such that a ViT can be trained from scratch by using only this subset of patches and achieve similar accuracy to the ViTs trained by using all image patches. We call this subset of input patches the winning tickets, which represent a significant amount of information in the input data. We use a ticket selector to generate the winning tickets based on the informativeness of patches for various types of ViT, including DeiT, LV-ViT, and Swin Transformers. The experiments show that there is a clear difference between the performance of models trained with winning tickets and randomly selected subsets, which verifies our proposed theory. We elaborate the analogical similarity between our proposed Data-LTH-ViTs and the conventional LTH for further verifying the integrity of our theory. The Source codes are available at https://github.com/shawnricecake/vit-lottery-ticket-input.

Authors

Geng Yuan

2 papers

Yanzhi Wang

3 papers

Xuan Shen

1 papers

TL;DR

Abstract

Authors

References51 items

Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training

Width & Depth Pruning for Vision Transformers

Unified Visual Transformer Compression

The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy

Dual Lottery Ticket Hypothesis

Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations

Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space

Multi-Dimensional Model Compression of Vision Transformer

SPViT: Enabling Faster Vision Transformers via Latency-Aware Soft Token Pruning

MIA-Former: Efficient and Robust Vision Transformers via Multi-grained Input-Adaptation

Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer

Rethinking and Improving Relative Position Encoding for Vision Transformer

Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?

AutoFormer: Searching Transformers for Visual Recognition

IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers

How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

BEiT: BERT Pre-Training of Image Transformers

Chasing Sparsity in Vision Transformers: An End-to-End Exploration

Patch Slimming for Efficient Vision Transformers

DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification

KVT: k-NN Attention for Boosting Vision Transformers

All Tokens Matter: Token Labeling for Training Better Vision Transformers

Scalable Vision Transformers with Hierarchical Pooling

Training data-efficient image transformers & distillation through attention

The Lottery Ticket Hypothesis for Object Recognition

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

The Lottery Ticket Hypothesis for Pre-trained BERT Networks

End-to-End Object Detection with Transformers

When BERT Plays the Lottery, All Tickets Are Winning

Comparing Rewinding and Fine-tuning in Neural Network Pruning

Linear Mode Connectivity and the Lottery Ticket Hypothesis

Evaluating Lottery Tickets Under Distributional Shifts

Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP

Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask

The State of Sparsity in Deep Neural Networks

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights

To prune, or not to prune: exploring the efficacy of pruning for model compression

Deep Residual Learning for Image Recognition

ImageNet: A large-scale hierarchical image database

Et al

WHO Technical Report

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Efficient Training of Visual Transformers with Small-Size Datasets

and Yanzhi Wang

In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

and Michael Carbin

pages 3259– 3269

Hongyuan Zhan

Learning Multiple Layers of Features from Tiny Images

In Advances in Neural Information Processing Systems

Field of Study

Venue Information

Name

Type

URL

Alternate Names