CAILA: Concept-Aware Intra-Layer Adapters for Compositional Zero-Shot Learning (2023-05-26T00:00:00.000000Z)

TL;DR

This paper proposes to dive deep into the architecture and insert adapters, a parameter-efficient technique proven to be effective among large language models, into each CLIP encoder layer, and equip adapters with concept awareness so that concept-specific features of "object", "attribute", and "composition" can be extracted.

Abstract

In this paper, we study the problem of Compositional Zero-Shot Learning (CZSL), which is to recognize novel attribute-object combinations with pre-existing concepts. Recent researchers focus on applying large-scale Vision-Language Pre-trained (VLP) models like CLIP with strong generalization ability. However, these methods treat the pre-trained model as a black box and focus on pre- and post-CLIP operations, which do not inherently mine the semantic concept between the layers inside CLIP. We propose to dive deep into the architecture and insert adapters, a parameter-efficient technique proven to be effective among large language models, into each CLIP encoder layer. We further equip adapters with concept awareness so that concept-specific features of "object", "attribute", and "composition" can be extracted. We assess our method on four popular CZSL datasets, MIT-States, C-GQA, UT-Zappos, and VAW-CZSL, which shows state-of-the-art performance compared to existing methods on all of them.

Authors

Haidong Zhu

2 papers

Zhao-Heng Zheng

2 papers

Ramkant Nevatia

2 papers

TL;DR

Abstract

Authors

References52 items

Decomposed Soft Prompt Guided Fusion Enhancing for Compositional Zero-Shot Learning

Learning Attention Propagation for Compositional Zero-Shot Learning

FashionVLP: Vision Language Transformer for Fashion Retrieval with Feedback

Siamese Contrastive Embedding Network for Compositional Zero-Shot Learning

Vision Transformer Adapter for Dense Predictions

Disentangling Visual Embeddings for Attributes and Objects

KG-SP: Knowledge Guided Simple Primitives for Open World Compositional Zero-Shot Learning

Learning to Compose Soft Prompts for Compositional Zero-Shot Learning

Conditional Prompt Learning for Vision-Language Models

VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

Scaling Up Vision-Language Pretraining for Image Captioning

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

Learning to Prompt for Vision-Language Models

FREE: Feature Refinement for Generalized Zero-Shot Learning

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation

Compacter: Efficient Low-Rank Hypercomplex Adapter Layers

Independent Prototype Propagation for Zero-Shot Compositionality

Learning Graph Embeddings for Open World Compositional Zero-Shot Learning

The Power of Scale for Parameter-Efficient Prompt Tuning

Learning Transferable Visual Models From Natural Language Supervision

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Learning Graph Embeddings for Compositional Zero-shot Learning

Open World Compositional Zero-Shot Learning

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

A causal view of compositional zero-shot recognition

Learning Unseen Concepts via Hierarchical Decomposition and Composition

Symmetry and Group in Attribute-Object Compositions

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Adversarial Fine-Grained Composition Learning for Unseen Attribute-Object Recognition

Recognizing Unseen Attribute-Object Pair with Generative Model

Task-Driven Modular Networks for Zero-Shot Compositional Learning

TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning

GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

Parameter-Efficient Transfer Learning for NLP

Rethinking Knowledge Graph Propagation for Zero-Shot Learning

Zero-Shot Recognition via Semantic Embeddings and Knowledge Graphs

Feature Generating Networks for Zero-Shot Learning

A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts

Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly

From Red Wine to Red Tomato: Composition with Context

Semantic Jitter: Dense Supervision for Visual Comparisons via Synthetic Images

Learning a Deep Embedding Model for Zero-Shot Learning

Semi-Supervised Classification with Graph Convolutional Networks

Learning Deep Representations of Fine-Grained Visual Descriptions

Deep Residual Learning for Image Recognition

Discovering states and transformations in image collections

GloVe: Global Vectors for Word Representation

Fine-Grained Visual Comparisons with Local Learning

Zero-Shot Learning Through Cross-Modal Transfer

Categorization

Attribute-Based Classification for Zero-Shot Visual Object Categorization

Zero-shot 1719 Authorized licensed use limited to the terms of the applicable

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names