DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations (2022-06-20T00:00:00.000000Z)

TL;DR

The strong alignment of textual and visual features pretrained with millions of auxiliary image-text pairs is utilized and Dual Context Optimization (DualCoOp) is proposed as a unified framework for partial-label MLR and zero-shot MLR.

Abstract

Solving multi-label recognition (MLR) for images in the low-label regime is a challenging task with many real-world applications. Recent work learns an alignment between textual and visual spaces to compensate for insufficient image labels, but loses accuracy because of the limited amount of available MLR annotations. In this work, we utilize the strong alignment of textual and visual features pretrained with millions of auxiliary image-text pairs and propose Dual Context Optimization (DualCoOp) as a unified framework for partial-label MLR and zero-shot MLR. DualCoOp encodes positive and negative contexts with class names as part of the linguistic input (i.e. prompts). Since DualCoOp only introduces a very light learnable overhead upon the pretrained vision-language framework, it can quickly adapt to multi-label recognition tasks that have limited annotations and even unseen classes. Experiments on standard multi-label recognition benchmarks across two challenging low-label settings demonstrate the advantages of our approach over state-of-the-art methods.

Authors

Kate Saenko

26 papers

Ping Hu

2 papers

Ximeng Sun

1 papers

TL;DR

Abstract

Authors

References87 items

Prompt Distribution Learning

Unsupervised Prompt Learning for Vision-Language Models

Conditional Prompt Learning for Vision-Language Models

Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels

Structured Semantic Transfer for Multi-Label Recognition with Partial Labels

Prompting Visual-Language Models for Efficient Video Understanding

Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models

Robust fine-tuning of zero-shot models

Learning to Prompt for Vision-Language Models

Discriminative Region-based Multi-Label Zero-Shot Learning

Residual Attention: A Simple but Effective Method for Multi-Label Recognition

Semantic Diversity Learning for Zero-Shot Multi-label Classification

The Power of Scale for Parameter-Efficient Prompt Tuning

Learning Transferable Visual Models From Natural Language Supervision

Understanding deep learning (still) requires rethinking generalization

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Generative Multi-Label Zero-Shot Learning

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Asymmetric Loss For Multi-Label Classification

Knowledge-Guided Multi-Label Few-Shot Learning for General Image Recognition

A Shared Multi-Attention Framework for Multi-Label Zero-Shot Learning

Interactive Multi-Label CNN Learning With Partial Labels

Deep0Tag: Deep Multiple Instance Learning for Zero-Shot Image Tagging

Orderless Recurrent Models for Multi-Label Classification

Multi-Label Classification with Label Graph Superimposing

Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition

Multi-Label Image Recognition With Graph Convolutional Networks

Learning a Deep ConvNet for Multi-Label Classification With Partial Labels

The Open Images Dataset V4

Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detection

Deep Imbalanced Attribute Classification using Visual Attention Aggregation

Bilinear Attention Networks

Exploring the Limits of Weakly Supervised Pretraining

Partial Multi-Label Learning

Zero-Shot Object Detection

Deep Multiple Instance Learning for Zero-shot Image Tagging

Multilabel Remote Sensing Image Retrieval Using a Semisupervised Graph-Theoretic Method

Multi-label Zero-Shot Learning with Structured Knowledge Graphs

Multi-label Image Recognition by Recurrently Discovering Attentional Regions

Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

Attention is All you Need

Learning Spatial Regularization with Image-Level Supervisions for Multi-label Image Classification

Learning Visual N-Grams from Web Data

Beyond Object Proposals: Random Crop Pooling for Multi-Label Image Recognition

Semantic Regularisation for Recurrent Image Annotation

Human Attribute Recognition by Deep Hierarchical Contexts

Fast Zero-Shot Image Tagging

CNN-RNN: A Unified Framework for Multi-label Image Classification

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels

Deep Residual Learning for Image Recognition

On the Optimality of Classifier Chain for Multi-label Classification

Learning Visual Features from Large Weakly Supervised Data

Label-Embedding for Image Classification

Very Deep Convolutional Networks for Large-Scale Image Recognition

COSTA: Co-Occurrence Statistics for Zero-Shot Classification

Microsoft COCO: Common Objects in Context

Zero-Shot Learning by Convex Combination of Semantic Embeddings

Fast Image Tagging

Multi-label learning with incomplete class assignments

Multi-Label Learning with Weak Label

NUS-WIDE: a real-world web image database from National University of Singapore

ImageNet: A large-scale hierarchical image database

Multi-Label Classification: An Overview

Prefix-Tuning: Optimizing Continuous Prompts for Generation

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

DenseCLIP: Extract Free Dense Labels from CLIP

Prompt-Based Multi-Modal Image Segmentation

Autoprompt: Eliciting knowledge from language models with automatically generated prompts

Language Models are Unsupervised Multitask Learners

International Journal of Computer Vision manuscript No. (will be inserted by the editor) The PASCAL Visual Object Classes (VOC) Challenge

Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content

c) Did you include any new assets either in the supplemental material or as a URL?

Did you discuss any potential negative societal impacts of your work? [Yes] Please refer to the Negative Societal Impacts