Zero-guidance Segmentation Using Zero Segment Labels (2023-03-23T00:00:00.000000Z)

TL;DR

This research proposes a novel problem zero-guidance segmentation and the first baseline that leverages two pre-trained generalist models, DINO and CLIP, to solve this problem without any fine-tuning or segmentation dataset.

Abstract

The joint visual-language model CLIP has enabled new and exciting applications, such as open-vocabulary segmentation, which can locate any segment given an arbitrary text query. In our research, we ask whether it is possible to discover semantic segments without any user guidance in the form of text queries or predefined classes, and label them using natural language automatically? We propose a novel problem zero-guidance segmentation and the first baseline that leverages two pre-trained generalist models, DINO and CLIP, to solve this problem without any fine-tuning or segmentation dataset. The general idea is to first segment an image into small over-segments, encode them into CLIP’s visual-language space, translate them into text labels, and merge semantically similar segments together. The key challenge, however, is how to encode a visual segment into a segment-specific embedding that balances global and local context information, both useful for recognition. Our main contribution is a novel attention-masking technique that balances the two contexts by analyzing the attention layers inside CLIP. We also introduce several metrics for the evaluation of this new task. With CLIP’s innate knowledge, our method can precisely locate the Mona Lisa painting among a museum crowd (Figure 1). More results are available at https://zero-guide-seg.github.io/.

Authors

Supasorn Suwajanakorn

2 papers

Pitchaporn Rewatbowornwong

1 papers

Nattanat Chatthee

1 papers

TL;DR

Abstract

Authors

References43 items

Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models

Imagic: Text-Based Real Image Editing with Diffusion Models

Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP

Unsupervised Semantic Segmentation by Distilling Feature Correspondences

Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut

GroupViT: Semantic Segmentation Emerges from Text Supervision

Language-driven Semantic Segmentation

Generalized Category Discovery

Decoupling Zero-Shot Semantic Segmentation

Deep ViT Features as Dense Visual Descriptors

Masked-attention Mask Transformer for Universal Image Segmentation

ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic

ClipCap: CLIP Prefix for Image Captioning

Localizing Objects with Self-Supervised Transformers and no Labels

How Much Can CLIP Benefit Vision-and-Language Tasks?

Emerging Properties in Self-Supervised Vision Transformers

Vision Transformers with Patch Diversification

CLIPScore: A Reference-free Evaluation Metric for Image Captioning

SimCSE: Simple Contrastive Learning of Sentence Embeddings

DeepViT: Towards Deeper Vision Transformer

Learning Transferable Visual Models From Natural Language Supervision

Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Tasks

Image Segmentation Using Deep Learning: A Survey

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Object Detection in 20 Years: A Survey

Referring Image Segmentation via Recurrent Refinement Networks

Rethinking Atrous Convolution for Semantic Image Segmentation

Attention is All you Need

Pyramid Scene Parsing Network

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

The Role of Context for Object Detection and Semantic Segmentation in the Wild

Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials

Context-aware saliency detection

Saliency Detection: A Spectral Residual Approach

Deeper vs Wider: A Revisit of Transformer Configuration

A Simple Baseline for Zero-shot Semantic Segmentation with Pre-trained Vision-language Model

Open-Vocabulary Image Segmentation

DenseCLIP: Extract Free Dense Labels from CLIP

On the sentence embeddings from pre-trained language models

Language Models are Unsupervised Multitask Learners

Semantic textual similarity-multilingual and cross-lingual focused evaluation

: Semantic image

International Journal of Computer Vision manuscript No. (will be inserted by the editor) The PASCAL Visual Object Classes (VOC) Challenge

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names