CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention (2022-09-28T00:00:00.000000Z)

TL;DR

A free-lunch enhancement method, CALIP, to boost CLIP's zero-shot performance via a parameter-free attention module and inserts a small number of linear layers in CALIP's attention module to verify the robustness under the few-shot settings.

Abstract

Contrastive Language-Image Pre-training (CLIP) has been shown to learn visual representations with promising zero-shot performance. To further improve its downstream accuracy, existing works propose additional learnable modules upon CLIP and fine-tune them by few-shot training sets. However, the resulting extra training cost and data requirement severely hinder the efficiency for model deployment and knowledge transfer. In this paper, we introduce a free-lunch enhancement method, CALIP, to boost CLIP's zero-shot performance via a parameter-free attention module. Specifically, we guide visual and textual representations to interact with each other and explore cross-modal informative features via attention. As the pre-training has largely reduced the embedding distances between two modalities, we discard all learnable parameters in the attention and bidirectionally update the multi-modal features, enabling the whole process to be parameter-free and training-free. In this way, the images are blended with textual-aware signals and the text representations become visual-guided for better adaptive zero-shot alignment. We evaluate CALIP on various benchmarks of 14 datasets for both 2D image and 3D point cloud few-shot classification, showing consistent zero-shot performance improvement over CLIP. Based on that, we further insert a small number of linear layers in CALIP's attention module and verify our robustness under the few-shot settings, which also achieves leading performance compared to existing methods. Those extensive experiments demonstrate the superiority of our approach for efficient enhancement of CLIP. Code is available at https://github.com/ZiyuGuo99/CALIP.

Authors

References54 items

Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimicking

Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners

Learning 3D Representations from 2D Pre-Trained Models via Image-to-Point Masked Autoencoders

A Multi-level Mesh Mutual Attention Model for Visual Question Answering

Collaboration of Pre-trained Models Makes Better Few-shot Learner

Frozen CLIP Models are Efficient Video Learners

Can Language Understand Depth?

Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training

Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model

UniFormer: Unifying Convolution and Self-Attention for Visual Recognition

PointCLIP: Point Cloud Understanding by CLIP

VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts

DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting

Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

ActionCLIP: A New Paradigm for Video Action Recognition

Learning to Prompt for Vision-Language Models

SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers

Dual-stream Network for Visual Recognition

Learning Transferable Visual Models From Natural Language Supervision

TrackFormer: Multi-Object Tracking with Transformers

PCT: Point cloud transformer

End-to-End Object Detection with Adaptive Clustering Transformer

The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

End-to-End Object Detection with Transformers

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Natural Adversarial Examples

Learning Robust Global Representations by Penalizing Local Predictive Power

Do ImageNet Classifiers Generalize to ImageNet?

Parameter-Efficient Transfer Learning for NLP

Image Transformer

EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification

Attention is All you Need

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Multi-view 3D Object Detection Network for Autonomous Driving

Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC)

Deep Residual Learning for Image Recognition

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Food-101 - Mining Discriminative Components with Random Forests

3D ShapeNets: A deep representation for volumetric shapes

3D Object Representations for Fine-Grained Categorization

Describing Textures in the Wild

Fine-Grained Visual Classification of Aircraft

ImageNet classification with deep convolutional neural networks

UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

SUN database: Large-scale scene recognition from abbey to zoo

ImageNet: A large-scale hierarchical image database

Automated Flower Classification over a Large Number of Classes

Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories

Care of aged doctors

PointCLIP V2: Adapting CLIP for Powerful 3D Open-world Learning

MonoDETR: Depth-aware Transformer for Monocular 3D Object Detection

Prefix-Tuning: Optimizing Continuous Prompts for Generation