CRIS: CLIP-Driven Referring Image Segmentation (2021-11-30T00:00:00.000000Z)

TL;DR

This paper designs a vision-language decoder to propagate fine-grained semantic information from textual representations to each pixel-level activation, which promotes consistency between the two modalities and presents text-to-pixel contrastive learning.

Abstract

Referring image segmentation aims to segment a referent via a natural linguistic expression. Due to the distinct data properties between text and image, it is challenging for a network to well align text and pixel-level features. Existing approaches use pretrained models to facilitate learning, yet separately transfer the language/vision knowledge from pretrained models, ignoring the multi-modal corresponding information. Inspired by the recent advance in Contrastive Language-Image Pretraining (CLIP), in this paper, we propose an end-to-end CLIP-Driven Referring Image Segmen-tation framework (CRIS). To transfer the multi-modal knowledge effectively, CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment. More specifically, we design a vision-language decoder to propagate fine-grained semantic information from textual representations to each pixel-level activation, which promotes consistency between the two modalities. In addition, we present text-to-pixel contrastive learning to explicitly enforce the text feature similar to the related pixel-level features and dissimilar to the irrelevances. The experimental results on three benchmark datasets demonstrate that our proposed framework significantly outperforms the state-of-the-art performance without any post-processing.

Authors

Tongliang Liu

8 papers

Xunqiang Tao

2 papers

Zhaoqing Wang

1 papers

TL;DR

Abstract

Authors

References48 items

Selective-Supervised Contrastive Learning with Noisy Labels

CLIP4Caption: CLIP for Video Caption

SimVLM: Simple Visual Language Model Pretraining with Weak Supervision

Vision-Language Transformer and Query Generation for Referring Segmentation

CLIP2Video: Mastering Video-Text Retrieval via Image CLIP

Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentation

CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery

Locate then Segment: A Strong Pipeline for Referring Image Segmentation

Learning Transferable Visual Models From Natural Language Supervision

Dense Contrastive Learning for Self-Supervised Visual Pre-Training

Unsupervised Learning of Dense Visual Representations

Cascade Grouped Attention Network for Referring Expression Segmentation

Linguistic Structure Guided Context Modeling for Referring Image Segmentation

GINet: Graph Interaction Network for Scene Parsing

Referring Image Segmentation via Cross-Modal Progressive Comprehension

Bi-Directional Relationship Inferring Network for Referring Image Segmentation

End-to-End Object Detection with Transformers

Multi-Task Collaborative Network for Joint Referring Expression Comprehension and Segmentation

Improved Baselines with Momentum Contrastive Learning

A Simple Framework for Contrastive Learning of Visual Representations

End-to-End Learning of Visual Representations From Uncurated Instructional Videos

Momentum Contrast for Unsupervised Visual Representation Learning

Referring Expression Object Segmentation with Caption-Aware Consistency

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

Adaptive Pyramid Context Network for Semantic Segmentation

Cross-Modal Self-Attention Network for Referring Image Segmentation

Expression

Learning to Assemble Neural Module Tree Networks for Visual Grounding

Dual Attention Network for Scene Segmentation

Key-Word-Aware Network for Referring Expression Image Segmentation

Dynamic Multimodal Instance Segmentation guided by natural language queries

An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution

Unsupervised Feature Learning via Non-parametric Instance Discrimination

MAttNet: Modular Attention Network for Referring Expression Comprehension

Attention is All you Need

Recurrent Multimodal Interaction for Referring Image Segmentation

Mask R-CNN

Layer Normalization

Segmentation from Natural Language Expressions

Deep Residual Learning for Image Recognition

Neural Machine Translation of Rare Words with Subword Units

Fully convolutional networks for semantic segmentation

ReferItGame: Referring to Objects in Photographs of Natural Scenes

Microsoft COCO: Common Objects in Context

Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials

Dimensionality Reduction by Learning an Invariant Mapping

Language Models are Unsupervised Multitask Learners

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names