Open-Vocabulary Universal Image Segmentation with MaskCLIP
Published in
International Conference on Machine Learning(2022)External Links:
TL
TL;DR
The developed MaskCLIP is an encoder-only module that seamlessly integrates mask tokens with a pre-trained ViT CLIP model for semantic/instance segmentation and class prediction that avoids the time-consuming student-teacher training process.
Abstract
In this paper, we tackle an emerging computer vision task, open-vocabulary universal image segmentation, that aims to perform semantic/instance/panoptic segmentation (background semantic labeling + foreground instance segmentation) for arbitrary categories of text-based descriptions in inference time. We first build a baseline method by directly adopting pre-trained CLIP models without finetuning or distillation. We then develop MaskCLIP, a Transformer-based approach with a MaskCLIP Visual Encoder, which is an encoder-only module that seamlessly integrates mask tokens with a pre-trained ViT CLIP model for semantic/instance segmentation and class prediction. MaskCLIP learns to efficiently and effectively utilize pre-trained partial/dense CLIP features within the MaskCLIP Visual Encoder that avoids the time-consuming student-teacher training process. MaskCLIP outperforms previous methods for semantic/instance/panoptic segmentation on ADE20K and PASCAL datasets. We show qualitative illustrations for MaskCLIP with online custom categories. Project website: https://maskclip.github.io.
References43 items
1
X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks
2
Open-Vocabulary DETR with Conditional Matching
3
CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
4
Language-driven Semantic Segmentation
5