TagCLIP: A Local-to-Global Framework to Enhance Open-Vocabulary Multi-Label Classification of CLIP Without Training (2023-12-20T00:00:00.000000Z)

TL;DR

This work dissects the preservation of patch-wise spatial information in CLIP and proposed a local-to-global framework to obtain image tags, which significantly enhances its multi-label classification performance on various benchmarks without dataset-specific training.

Abstract

Contrastive Language-Image Pre-training (CLIP) has demonstrated impressive capabilities in open-vocabulary classification. The class token in the image encoder is trained to capture the global features to distinguish different text descriptions supervised by contrastive loss, making it highly effective for single-label classification. However, it shows poor performance on multi-label datasets because the global feature tends to be dominated by the most prominent class and the contrastive nature of softmax operation aggravates it. In this study, we observe that the multi-label classification results heavily rely on discriminative local features but are overlooked by CLIP. As a result, we dissect the preservation of patch-wise spatial information in CLIP and proposed a local-to-global framework to obtain image tags. It comprises three steps: (1) patch-level classification to obtain coarse scores; (2) dual-masking attention refinement (DMAR) module to refine the coarse scores; (3) class-wise reidentification (CWR) module to remedy predictions from a global perspective. This framework is solely based on frozen CLIP and significantly enhances its multi-label classification performance on various benchmarks without dataset-specific training. Besides, to comprehensively assess the quality and practicality of generated tags, we extend their application to the downstream task, i.e., weakly supervised semantic segmentation (WSSS) with generated tags as image-level pseudo labels. Experiments demonstrate that this classify-then-segment paradigm dramatically outperforms other annotation-free segmentation methods and validates the effectiveness of generated tags. Our code is available at https://github.com/linyq2117/TagCLIP.

References48 items

A closer look at the explainability of Contrastive language-image pre-training

What do Vision Transformers Learn? A Visual Exploration

SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation

Texts as Images in Prompt Tuning for Multi-Label Image Recognition

Open-Vocabulary Multi-Label Classification via Multi-modal Knowledge Transfer

DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations

ReCo: Retrieve and Co-segment for Zero-shot Transfer

Discovering Object Masks with Transformers for Unsupervised Semantic Segmentation

Self-Supervised Learning of Object Parts for Semantic Segmentation

VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance

Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers

Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels

GroupViT: Semantic Segmentation Emerges from Text Supervision

Debiased Learning from Naturally Imbalanced Pseudo-Labels

RegionCLIP: Region-based Language-Image Pretraining

Extract Free Dense Labels from CLIP

Discriminative Region-based Multi-Label Zero-Shot Learning

Do Vision Transformers See Like Convolutional Neural Networks?

Semantic Diversity Learning for Zero-Shot Multi-label Classification

Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering

TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization

Learning Transferable Visual Models From Natural Language Supervision

Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals

Attention-Driven Dynamic Graph Convolutional Network for Multi-label Image Recognition

Asymmetric Loss For Multi-Label Classification

Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets

Rethinking Pre-training and Self-training

A Shared Multi-Attention Framework for Multi-Label Zero-Shot Learning

Self-Supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation

Cross-Modality Attention with Semantic Graph Embedding for Multi-Label Classification

Self-Training With Noisy Student Improves ImageNet Classification

SegSort: Segmentation by Discriminative Sorting of Segments

Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition

Multi-Label Image Recognition With Graph Convolutional Networks

Invariant Information Clustering for Unsupervised Image Classification and Segmentation

Learning Pixel-Level Semantic Affinity with Image-Level Supervision for Weakly Supervised Semantic Segmentation

Multi-label Image Recognition by Recurrently Discovering Attentional Regions

COCO-Stuff: Thing and Stuff Classes in Context

Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

Learning Deep Features for Discriminative Localization

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Microsoft COCO: Common Objects in Context

International Journal of Computer Vision manuscript No. (will be inserted by the editor) The PASCAL Visual Object Classes (VOC) Challenge

2022. Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering Transform-ers

2022a. Namedmask: Dis-tilling segmenters from complementary foundation models

2022. Learning to prompt for vision-language models