Side Adapter Network for Open-Vocabulary Semantic Segmentation (2023-02-23T00:00:00.000000Z)

TL;DR

This paper presents a new framework for open-vocabulary semantic segmentation with the pre-trained vision-language model, named Side Adapter Network (SAN), which significantly outperforms other counterparts, with up to 18 times fewer trainable parameters and 19 times faster inference speed.

Abstract

This paper presents a new framework for open-vocabulary semantic segmentation with the pre-trained vision-language model, named Side Adapter Network (SAN). Our approach models the semantic segmentation task as a region recognition problem. A side network is attached to a frozen CLIP model with two branches: one for predicting mask proposals, and the other for predicting attention bias which is applied in the CLIP model to recognize the class of masks. This decoupled design has the benefit CLIP in recognizing the class of mask proposals. Since the attached side network can reuse CLIP features, it can be very light. In addition, the entire network can be trained end-to-end, allowing the side network to be adapted to the frozen CLIP model, which makes the predicted mask proposals CLIP-aware. Our approach is fast, accurate, and only adds a few additional trainable parameters. We evaluate our approach on multiple semantic segmentation benchmarks. Our method significantly outperforms other counterparts, with up to 18 times fewer trainable parameters and 19 times faster inference speed. Fig. 1 shows some visualization results on ImageNet. We hope our approach will serve as a solid baseline and help ease future research in open-vocabulary semantic segmentation.

Authors

Han Hu

9 papers

Fangyun Wei

10 papers

Xiang Bai

6 papers

TL;DR

Abstract

Authors

References44 items

Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP

LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning

CoCa: Contrastive Captioners are Image-Text Foundation Models

Flamingo: a Visual Language Model for Few-Shot Learning

GroupViT: Semantic Segmentation Emerges from Text Supervision

Language-driven Semantic Segmentation

A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model

Scaling Open-Vocabulary Image Segmentation with Image-Level Labels

Decoupling Zero-Shot Semantic Segmentation

VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

Masked-attention Mask Transformer for Universal Image Segmentation

CRIS: CLIP-Driven Referring Image Segmentation

Florence: A New Foundation Model for Computer Vision

Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

Learning to Prompt for Vision-Language Models

Per-Pixel Classification is Not All You Need for Semantic Segmentation

Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

CLIPScore: A Reference-free Evaluation Metric for Image Captioning

The Power of Scale for Parameter-Efficient Prompt Tuning

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery

Learning Transferable Visual Models From Natural Language Supervision

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks

UNITER: Learning UNiversal Image-TExt Representations

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Zero-Shot Semantic Segmentation

Semantic Projection Network for Zero- and Few-Label Semantic Segmentation

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Parameter-Efficient Transfer Learning for NLP

Scene Parsing through ADE20K Dataset

Open Vocabulary Scene Parsing

COCO-Stuff: Thing and Stuff Classes in Context

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

Deep Residual Learning for Image Recognition

VQA: Visual Question Answering

Fully convolutional networks for semantic segmentation

The Role of Context for Object Detection and Semantic Segmentation in the Wild

Open-Vocabulary Panoptic Segmentation with MaskCLIP

Authorized licensed use limited to the terms of the applicable license agreement with IEEE

Prefix-Tuning: Optimizing Continuous Prompts for Generation

DenseCLIP: Extract Free Dense Labels from CLIP

The PASCAL Visual Object Classes Challenge

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names