Towards Robust Vision Transformer (2021-05-17T00:00:00.000000Z)

TL;DR

Robust Vision Transformer (RVT) is proposed, which is a new vision transformer and has superior performance with strong robustness and generalization ability compared with previous ViTs and state-of-the-art CNNs.

Abstract

Recent advances on Vision Transformer (ViT) and its improved variants have shown that self-attention-based networks surpass traditional Convolutional Neural Networks (CNNs) in most vision tasks. However, existing ViTs focus on the standard accuracy and computation cost, lacking the investigation of the intrinsic influence on model robustness and generalization. In this work, we conduct systematic evaluation on components of ViTs in terms of their impact on robustness to adversarial examples, common corruptions and distribution shifts. We find some components can be harmful to robustness. By leveraging robust components as building blocks of ViTs, we propose Robust Vision Transformer (RVT), which is a new vision transformer and has superior performance with strong robustness. Inspired by the findings during the evaluation, we further propose two new plug-and-play techniques called position-aware attention scaling and patch-wise augmentation to augment our RVT, which we abbreviate as RVT*. The experimental results of RVT on ImageNet and six robustness benchmarks demonstrate its advanced robustness and generalization ability compared with previous ViTs and state-of-the-art CNNs. Furthermore, RVT-S* achieves Top-1 rank on multiple robustness leaderboards including ImageNet-C, ImageNet-Sketch and ImageNet-R.

Authors

Hui Xue

4 papers

Shaokai Ye

3 papers

Yuan He

3 papers

TL;DR

Abstract

Authors

References52 items

Are Transformers More Robust Than CNNs?

Vision Transformers are Robust Learners

LocalViT: Analyzing Locality in Vision Transformers

LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference

On the Robustness of Vision Transformers to Adversarial Examples

Rethinking Spatial Dimensions of Vision Transformers

CvT: Introducing Convolutions to Vision Transformers

Understanding Robustness of Transformers for Image Classification

Incorporating Convolution Designs into Visual Transformers

ConViT: improving vision transformers with soft convolutional inductive biases

Transformer in Transformer

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

Training data-efficient image transformers & distillation through attention

An Effective Anti-Aliasing Approach for Residual Networks

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Shape-Texture Debiased Neural Network Training

Do Wider Neural Networks Really Help Adversarial Robustness?

Adversarially Robust Neural Architectures

The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

Rethinking Positional Encoding in Language Pre-training

Smooth Adversarial Training

Designing Network Design Spaces

A Simple Way to Make Neural Networks Robust Against Diverse Image Corruptions

AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty

When NAS Meets Robustness: In Search of Robust Architectures Against Adversarial Attacks

Self-Training With Noisy Student Improves ImageNet Classification

Kornia: an Open Source Differentiable Computer Vision Library for PyTorch

Randaugment: Practical automated data augmentation with a reduced search space

Natural Adversarial Examples

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Are Sixteen Heads Really Better than One?

Learning Robust Global Representations by Penalizing Local Predictive Power

Making Convolutional Networks Shift-Invariant Again

Defensive Quantization: When Efficiency Meets Robustness

Adversarial Robustness vs. Model Compression, or Both?

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Is Robustness the Cost of Accuracy? - A Comprehensive Study on the Robustness of 18 Deep Image Classification Models

Self-Attention with Relative Position Representations

mixup: Beyond Empirical Risk Minimization

Towards Deep Learning Models Resistant to Adversarial Attacks

Aggregated Residual Transformations for Deep Neural Networks

Inductive Bias of Deep Convolutional Networks through Pooling Geometry

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Explaining and Harnessing Adversarial Examples

Intriguing properties of neural networks

ImageNet: A large-scale hierarchical image database

Do We Really Need Explicit Position Encodings for Vision Transformers?

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Improving Language Understanding by Generative Pre-Training

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names