Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models (2023-03-08T00:00:00.000000Z)

TL;DR

ODISE is presented, which unifies pre-trained text-image diffusion and discriminative models to perform open-vocabulary panoptic segmentation and outperforms the previous state of the art by significant margins on both open- VocabularyPanoptic and semantic segmentation tasks.

Abstract

We present ODISE: Open-vocabulary DIffusion-based panoptic SEgmentation, which unifies pre-trained text-image diffusion and discriminative models to perform open-vocabulary panoptic segmentation. Text-to-image diffusion models have the remarkable ability to generate high-quality images with diverse open-vocabulary language descriptions. This demonstrates that their internal representation space is highly correlated with open concepts in the real world. Text-image discriminative models like CLIP, on the other hand, are good at classifying images into open-vocabulary labels. We leverage the frozen internal representations of both these models to perform panoptic segmentation of any category in the wild. Our approach outperforms the previous state of the art by significant margins on both open-vocabulary panoptic and semantic segmentation tasks. In particular, with COCO training only, our method achieves 23.4 PQ and 30.0 mIoU on the ADE20K dataset, with 8.3 PQ and 7.9 mIoU absolute improvement over the previous state of the art. We open-source our code and models at https://github.com/NVlabs/ODISE.

Authors

Arash Vahdat

3 papers

Wonmin Byeon

3 papers

Jiarui Xu

4 papers

TL;DR

Abstract

Authors

References105 items

eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers

Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Simple Open-Vocabulary Object Detection with Vision Transformers

DeiT III: Revenge of the ViT

Hierarchical Text-Conditional Image Generation with CLIP Latents

ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension

Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model

Open-Vocabulary DETR with Conditional Matching

GroupViT: Semantic Segmentation Emerges from Text Supervision

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations

A ConvNet for the 2020s

Language-driven Semantic Segmentation

Detecting Twenty-thousand Classes using Image-level Supervision

SLIP: Self-supervision meets Language-Image Pre-training

High-Resolution Image Synthesis with Latent Diffusion Models

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

RegionCLIP: Region-based Language-Image Pretraining

Decoupling Zero-Shot Semantic Segmentation

Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

Grounded Language-Image Pre-training

Label-Efficient Semantic Segmentation with Diffusion Models

Masked-attention Mask Transformer for Universal Image Segmentation

MViTv2: Improved Multiscale Vision Transformers for Classification and Detection

Towards Language-Free Training for Text-to-Image Generation

Masked Autoencoders Are Scalable Vision Learners

LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs

Ego4D: Around the World in 3,000 Hours of Egocentric Video

Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers

Learning Open-World Object Proposals without Learning to Classify

SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations

Per-Pixel Classification is Not All You Need for Semantic Segmentation

K-Net: Towards Unified Image Segmentation

Score-based Generative Modeling in Latent Space

REFINE: Prediction Fusion Network for Panoptic Segmentation

Diffusion Models Beat GANs on Image Synthesis

Emerging Properties in Self-Supervised Vision Transformers

Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort

Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation

An Empirical Study of Training Self-Supervised Vision Transformers

Repurposing GANs for One-Shot Semantic Part Segmentation

Learning Transferable Visual Models From Natural Language Supervision

Improved Denoising Diffusion Probabilistic Models

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Taming Transformers for High-Resolution Image Synthesis

Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation

Fully Convolutional Networks for Panoptic Segmentation

MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers

Score-Based Generative Modeling through Stochastic Differential Equations

Open-Vocabulary Object Detection Using Captions

Denoising Diffusion Implicit Models

Adversarial score matching and improved sampling for image generation

Denoising Diffusion Probabilistic Models

Learning High-Resolution Domain-Specific Representations with a GAN Generator

Contrastive Learning for Weakly Supervised Phrase Grounding

Improved Techniques for Training Score-Based Generative Models

End-to-End Object Detection with Transformers

Analyzing and Improving the Image Quality of StyleGAN

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation

Generative Modeling by Estimating Gradients of the Data Distribution

Zero-Shot Semantic Segmentation

LVIS: A Dataset for Large Vocabulary Instance Segmentation

Semantic Projection Network for Zero- and Few-Label Semantic Segmentation

Panoptic Feature Pyramid Networks

A Style-Based Generator Architecture for Generative Adversarial Networks

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Panoptic Segmentation

Decoupled Weight Decay Regularization

The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes