ResNet strikes back: An improved training procedure in timm (2021-10-01T00:00:00.000000Z)

TL;DR

This paper re-evaluate the performance of the vanilla ResNet-50 when trained with a procedure that integrates novel optimization and data-augmentation advances, and shares competitive training settings and pre-trained models in the timm open-source library, with the hope that they will serve as better baselines for future work.

Abstract

The influential Residual Networks designed by He et al. remain the gold-standard architecture in numerous scientific publications. They typically serve as the default architecture in studies, or as baselines when new architectures are proposed. Yet there has been significant progress on best practices for training neural networks since the inception of the ResNet architecture in 2015. Novel optimization&data-augmentation have increased the effectiveness of the training recipes. In this paper, we re-evaluate the performance of the vanilla ResNet-50 when trained with a procedure that integrates such advances. We share competitive training settings and pre-trained models in the timm open-source library, with the hope that they will serve as better baselines for future work. For instance, with our more demanding training setting, a vanilla ResNet-50 reaches 80.4% top-1 accuracy at resolution 224x224 on ImageNet-val without extra data or distillation. We also report the performance achieved with popular models with our training procedure.

Authors

Herv'e J'egou

7 papers

Hugo Touvron

5 papers

Ross Wightman

3 papers

TL;DR

Abstract

Authors

References62 items

Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision

ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training

MLP-Mixer: An all-MLP Architecture for Vision

Emerging Properties in Self-Supervised Vision Transformers

EfficientNetV2: Smaller Models and Faster Training

Going deeper with Image Transformers

Scaling Local Self-Attention for Parameter Efficient Visual Backbones

Revisiting ResNets: Improved Training and Scaling Strategies

Fast and Accurate Model Scaling

High-Performance Large-Scale Image Recognition Without Normalization

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

Training data-efficient image transformers & distillation through attention

Grafit: Learning fine-grained image representations with coarse labels

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Sharpness-Aware Minimization for Efficiently Improving Generalization

Slowing Down the Weight Norm Increase in Momentum-based Optimizers

Are we done with ImageNet?

Augment Your Batch: Improving Generalization Through Instance Repetition

Supervised Contrastive Learning

ResNeSt: Split-Attention Networks

TResNet: High Performance GPU-Dedicated Architecture

Designing Network Design Spaces

Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network

AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty

Randaugment: Practical automated data augmentation with a reduced search space

Fixing the train-test resolution discrepancy

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

Billion-scale semi-supervised learning for image classification

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

MultiGrain: a unified image embedding for classes and instances

Do ImageNet Classifiers Generalize to ImageNet?

Bag of Tricks for Image Classification with Convolutional Neural Networks

AutoAugment: Learning Augmentation Policies from Data

Averaging Weights Leads to Wider Optima and Better Generalization

Regularized Evolution for Image Classifier Architecture Search

Fixing Weight Decay Regularization in Adam

mixup: Beyond Empirical Risk Minimization

Squeeze-and-Excitation Networks

Random Erasing Data Augmentation

Learning Transferable Architectures for Scalable Image Recognition

The iNaturalist Species Classification and Detection Dataset

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Aggregated Residual Transformations for Deep Neural Networks

Deep Networks with Stochastic Depth

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Going deeper with convolutions

ImageNet Large Scale Visual Recognition Challenge

3D Object Representations for Fine-Grained Categorization

On the importance of initialization and momentum in deep learning

ImageNet classification with deep convolutional neural networks

ImageNet: A large-scale hierarchical image database

Automated Flower Classification over a Large Number of Classes

Acceleration of stochastic approximation by averaging

Pytorch image models

Dropout: a simple way to prevent neural networks from overfitting

Learning Multiple Layers of Features from Tiny Images

batchwise CutMix vs Mixup selection, lambda and CutMix region sampling performed per-batch

Procedure A2 is a 300 epochs schedule that is comparable to several modern procedures like DeiT, except with a larger batch size of 2048 and other choices

pairwise mixing, lambda, and region sampling performed per mixing sample pair within batch

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names