Open-vocabulary Attribute Detection (2022-11-23T00:00:00.000000Z)

TL;DR

The Open- Vocabulary Attribute Detection (OVAD) task and the corresponding OVAD benchmark are introduced to probe object-level attribute information learned by vision-language models and a first baseline method for open-vocabulary attribute detection is provided.

Abstract

Vision-language modeling has enabled open-vocabulary tasks where predictions can be queried using any text prompt in a zero-shot manner. Existing open-vocabulary tasks focus on object classes, whereas research on object attributes is limited due to the lack of a reliable attribute-focused evaluation benchmark. This paper introduces the Open- Vocabulary Attribute Detection (OVAD) task and the corresponding OVAD benchmark. The objective of the novel task and benchmark is to probe object-level attribute information learned by vision-language models. To this end, we created a clean and densely annotated test set covering 117 attribute classes on the 80 object classes of MS COCO. It includes positive and negative annotations, which enables open-vocabulary evaluation. Overall, the benchmark consists of 1.4 million annotations. For reference, we provide a first baseline method for open-vocabulary attribute detection. Moreover, we demonstrate the benchmark's value by studying the attribute detection performance of several foundation models.

Authors

T. Brox

22 papers

Sudhanshu Mittal

3 papers

M. A. Bravo

2 papers

TL;DR

Abstract

Authors

References68 items

LAION-5B: An open large-scale dataset for training next generation image-text models

Exploiting Unlabeled Data with Vision and Language Models for Object Detection

Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection

Localized Vision-Language Matching for Open-vocabulary Object Detection

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

Detecting Twenty-thousand Classes using Image-level Supervision

RegionCLIP: Region-based Language-Image Pretraining

Open Vocabulary Object Detection with Pseudo Bounding-Box Labels

Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

An Empirical Study of Training End-to-End Vision-and-Language Transformers

LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation

Learning to Predict Visual Attributes in the Wild

Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

Learning Transferable Visual Models From Natural Language Supervision

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

Open-Vocabulary Object Detection Using Captions

Learning to Infer Unseen Single-/ Multi-Attribute-Object Compositions With Graph Networks

Attribute Prototype Network for Zero-Shot Learning

Symmetry and Group in Attribute-Object Compositions

Locality and compositionality in zero-shot learning

Objects365: A Large-Scale, High-Quality Dataset for Object Detection

Gender

LVIS: A Dataset for Large Vocabulary Instance Segmentation

Maturity.

Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

The N-pattern

Zero-Shot Object Detection

Automatic Spatially-Aware Fashion Concept Discovery

Zero-Shot Learning — The Good, the Bad and the Ugly

Weight-averaged consistency targets improve semi-supervised deep learning results

Learning from Noisy Large-Scale Datasets with Minimal Supervision

COCO Attributes: Attributes for People, Animals, and Objects

Modeling Context in Referring Expressions

Recovering the Missing Link: Predicting Class-Attribute Associations for Unsupervised Zero-Shot Learning

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Deep Residual Learning for Image Recognition

Discovering states and transformations in image collections

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

VQA: Visual Question Answering

Microsoft COCO Captions: Data Collection and Evaluation Server

Fine-Grained Visual Comparisons with Local Learning

Microsoft COCO: Common Objects in Context

Im2Text: Describing Images Using 1 Million Captioned Photographs

Steven Bird, Ewan Klein and Edward Loper: Natural Language Processing with Python, Analyzing Text with the Natural Language Toolkit

Caltech-UCSD Birds 200

Automatic Attribute Discovery and Characterization from Noisy Web Data

Describing objects by their attributes

ImageNet: A large-scale hierarchical image database

A Scalable Hierarchical Distributed Language Model

Learning Visual Attributes

spacy: Industrial-strength natural language processing in python

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

A Survey of WordNets and their Licenses

license agreement with IEEE. Restrictions apply

International Journal of Computer Vision manuscript No. (will be inserted by the editor) The PASCAL Visual Object Classes (VOC) Challenge

color, clothes color, hair color: (black, white, gray, tan, brown, green, red, yellow, blue, orange, violet, pink) - This attribute refers to the visible color of the object

hair type - This attribute type refers to the hair type of the person and is classiﬁed either as curly/curled or straight

to objects which carry or comprise multiple elements or parts

Assign ‘unknown’ for cases where: (a) the attribute is not visible, like in the presence of occlusion or (b) a discrete label can not be assigned because of ambiguity or an in-between case

cooked - This attribute type is marked only for food object categories

Given an image, check for every object marked and verify that it has the correct class

Add bounding boxes for the missing objects, revise inaccurate bounding boxes

c) Annotate attributes as positive only based on their visual appearance

q) state - This attribute type is a non-exclusive attribute that contains multiple attribute sub-types like dry/wet, closed/open, turned on/

Edinburgh Research Explorer The PASCAL Visual Object Classes (VOC) Challenge

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL