ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy (2023-11-15T00:00:00.000000Z)

TL;DR

Although the selected models have similar ImageNet accuracies and compute requirements, it is found that they differ in many other aspects: types of mistakes, output calibration, transferability, and feature invariance, among others.

Abstract

Modern computer vision offers a great variety of models to practitioners, and selecting a model from multiple options for specific applications can be challenging. Conventionally, competing model architectures and training protocols are compared by their classification accuracy on ImageNet. However, this single metric does not fully capture performance nuances critical for specialized tasks. In this work, we conduct an in-depth comparative analysis of model behaviors beyond ImageNet accuracy, for both ConvNet and Vision Transformer architectures, each across supervised and CLIP training paradigms. Although our selected models have similar ImageNet accuracies and compute requirements, we find that they differ in many other aspects: types of mistakes, output calibration, transferability, and feature invariance, among others. This diversity in model characteristics, not captured by traditional metrics, highlights the need for more nuanced analysis when choosing among different models. Our code is available at https://github.com/kirill-vish/Beyond-INet.

Authors

Zhuang Liu

8 papers

Kirill Vishniakov

1 papers

Zhiqiang Shen

1 papers

TL;DR

Abstract

Authors

References65 items

Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks

PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning

Does Progress On Object Recognition Benchmarks Improve Real-World Generalization?

On the Connection between Pre-training Data Diversity and Fine-tuning Robustness

StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners

What Do Self-Supervised Vision Transformers Learn?

DataComp: In search of the next generation of multimodal datasets

Objectives Matter: Understanding the Impact of Self-Supervised Objectives on Vision Transformer Representations

Synthetic Data from Diffusion Models Improves ImageNet Classification

ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification

Scaling Vision Transformers to 22 Billion Parameters

Does progress on ImageNet transfer to real-world datasets?

ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders

Teaching Matters: Investigating the Role of Supervision in Vision Transformers

CLIPascene: Scene Sketching with Different Types and Levels of Abstraction

Finding Differences Between Transformers and ConvNets Using Counterfactual Simulation Testing

ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations

Is synthetic data from generative models ready for image recognition?

An Impartial Take to the CNN vs Transformer Robustness Contest

Can CNNs Be More Robust Than Transformers?

Beyond Supervised vs. Unsupervised: Representative Benchmarking and Analysis of Image Representation Learning

DeiT III: Revenge of the ViT

Hierarchical Text-Conditional Image Generation with CLIP Latents

Does Robustness on ImageNet Transfer to Downstream Tasks?

A ConvNet for the 2020s

Masked Autoencoders Are Scalable Vision Learners

Are Transformers More Robust Than CNNs?

Do Self-Supervised and Supervised Methods Learn Similar Visual Representations?

Robust fine-tuning of zero-shot models

Do Vision Transformers See Like Convolutional Neural Networks?

ConvNets vs. Transformers: Whose Visual Representations are More Transferable?

Accuracy on the Line: on the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization

Revisiting the Calibration of Modern Neural Networks

Intriguing Properties of Vision Transformers

Are Convolutional Neural Networks or Transformers more like human vision?

CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval

Learning Transferable Visual Models From Natural Language Supervision

Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels

Training data-efficient image transformers & distillation through attention

Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

On Robustness and Transferability of Convolutional Neural Networks

The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

Are we done with ImageNet?

From ImageNet to Image Classification: Contextualizing Progress on Benchmarks

Shortcut learning in deep neural networks

A Simple Framework for Contrastive Learning of Visual Representations

Big Transfer (BiT): General Visual Representation Learning

Momentum Contrast for Unsupervised Visual Representation Learning

A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark

Natural Adversarial Examples

Learning Robust Global Representations by Penalizing Local Predictive Power

Making Convolutional Networks Shift-Invariant Again

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Do ImageNet Classifiers Generalize to ImageNet?

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Why do deep convolutional networks generalize so poorly to small image transformations?

Do Better ImageNet Models Transfer Better?

On Calibration of Modern Neural Networks

Layer Normalization

Deep Residual Learning for Image Recognition

ImageNet classification with deep convolutional neural networks

ImageNet: A large-scale hierarchical image database

ImageNet suffers from dichotomous data difﬁculty

Gradient-based learning applied to document recognition

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names