GroupViT: Semantic Segmentation Emerges from Text Supervision (2022-02-22T00:00:00.000000Z)

TL;DR

A hierarchical Grouping Vision Transformer (GroupViT), which goes beyond the regular grid structure representation and learns to group image regions into progressively larger arbitrary-shaped segments and performs competitively to state-of-the-art transfer-learning methods requiring greater levels of supervision.

Abstract

Grouping and recognition are important components of visual scene understanding, e.g., for object detection and semantic segmentation. With end-to-end deep learning systems, grouping of image regions usually happens implicitly via top-down supervision from pixel-level recognition labels. Instead, in this paper, we propose to bring back the grouping mechanism into deep networks, which allows semantic segments to emerge automatically with only text supervision. We propose a hierarchical Grouping Vision Transformer (GroupViT), which goes beyond the regular grid structure representation and learns to group image regions into progressively larger arbitrary-shaped segments. We train GroupViT jointly with a text encoder on a large-scale image-text dataset via contrastive losses. With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i.e., without any further fine-tuning. It achieves a zero-shot accuracy of 52.3% mIoU on the PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context datasets, and performs competitively to state-of-the-art transfer-learning methods requiring greater levels of supervision. We open-source our code at https://github.com/NVlabs/GroupViT.

Authors

X. Wang

4 papers

Wonmin Byeon

3 papers

Thomas Breuel

2 papers

TL;DR

Abstract

Authors

References103 items

Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examples

RedCaps: web-curated image-text data created by the people, for the people

Pseudo-mask Matters in Weakly-supervised Semantic Segmentation

Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer

Video Swin Transformer

TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?

XCiT: Cross-Covariance Image Transformers

SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers

MLP-Mixer: An all-MLP Architecture for Vision

Emerging Properties in Self-Supervised Vision Transformers

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

MDETR - Modulated Detection for End-to-End Multi-Modal Understanding

Multiscale Vision Transformers

All you need are a few pixels: semantic segmentation with PixelPick

An Empirical Study of Training Self-Supervised Vision Transformers

Going deeper with Image Transformers

Boundary IoU: Improving Object-Centric Image Segmentation Evaluation

Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding

Mining Latent Classes for Few-shot Segmentation

ViViT: A Video Vision Transformer

CvT: Introducing Convolutions to Vision Transformers

Relation-aware Instance Refinement for Weakly Supervised Visual Grounding

Learning Transferable Visual Models From Natural Language Supervision

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Is Space-Time Attention All You Need for Video Understanding?

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

Puzzle-CAM: Improved Localization Via Matching Partial And Full Features

Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Training data-efficient image transformers & distillation through attention

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

DEAL: Difficulty-aware Active Learning for Semantic Segmentation

Contrastive Learning of Medical Visual Representations from Paired Images and Text

Prior Guided Feature Enrichment Network for Few-Shot Segmentation

Part-aware Prototype Network for Few-shot Semantic Segmentation

Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation

Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation

Object-Centric Learning with Slot Attention

Contrastive Learning for Weakly Supervised Phrase Grounding

VirTex: Learning Visual Representations from Textual Annotations

Large-Scale Adversarial Training for Vision-and-Language Representation Learning

Weakly-Supervised Semantic Segmentation via Sub-Category Exploration

Learning Integral Objects With Intra-Class Discriminator for Weakly-Supervised Semantic Segmentation

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Self-Supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation

Reinforced active learning for image segmentation

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Self-Supervised Difference Detection for Weakly-Supervised Semantic Segmentation

Feature Weighting and Boosting for Few-Shot Segmentation

UNITER: Learning UNiversal Image-TExt Representations

Unified Vision-Language Pre-Training for Image Captioning and VQA

Knowledge-guided Pairwise Reconstruction Network for Weakly Supervised Referring Expression Grounding

VisualBERT: A Simple and Performant Baseline for Vision and Language

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Zero-Shot Semantic Segmentation

Variational Adversarial Active Learning

FickleNet: Weakly and Semi-Supervised Semantic Image Segmentation Using Stochastic Inference

From Recognition to Cognition: Visual Commonsense Reasoning

Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

Unsupervised Textual Grounding: Linking Words to Image Concepts

Learning Pixel-Level Semantic Affinity with Image-Level Supervision for Weakly Supervised Semantic Segmentation

Knowledge Aided Consistency for Weakly Supervised Phrase Grounding

Conditional Image-Text Embedding Networks

Neural Discrete Representation Learning

Active Learning for Convolutional Neural Networks: A Core-Set Approach

Attention is All you Need

Pyramid Scene Parsing Network

Categorical Reparameterization with Gumbel-Softmax

The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables

SGDR: Stochastic Gradient Descent with Warm Restarts

Layer Normalization

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

The three R's of computer vision: Recognition, reconstruction and reorganization

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Learning Visual Features from Large Weakly Supervised Data