PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model (2024-03-21T00:00:00.000000Z)

TL;DR

Through extensive experiments, PSALM demonstrates its potential to transform the domain of image segmentation, leveraging the robust visual understanding capabilities of LMMs as seen in natural language processing.

Abstract

PSALM is a powerful extension of the Large Multi-modal Model (LMM) to address the segmentation task challenges. To overcome the limitation of the LMM being limited to textual output, PSALM incorporates a mask decoder and a well-designed input schema to handle a variety of segmentation tasks. This schema includes images, task instructions, conditional prompts, and mask tokens, which enable the model to generate and classify segmentation masks effectively. The flexible design of PSALM supports joint training across multiple datasets and tasks, leading to improved performance and task generalization. PSALM achieves superior results on several benchmarks, such as RefCOCO/RefCOCO+/RefCOCOg, COCO Panoptic Segmentation, and COCO-Interactive, and further exhibits zero-shot capabilities on unseen tasks, such as open-vocabulary segmentation, generalized referring expression segmentation and video object segmentation, making a significant step towards a GPT moment in computer vision. Through extensive experiments, PSALM demonstrates its potential to transform the domain of image segmentation, leveraging the robust visual understanding capabilities of LMMs as seen in natural language processing. Code and models are available at https://github.com/zamling/PSALM.

Authors

Zheng Zhang

1 papers

Yeyao Ma

1 papers

Enming Zhang

1 papers

TL;DR

Abstract

Authors

References66 items

OMG-Seg: Is One Model Good Enough for all Segmentation?

An Improved Baseline for Reasoning Segmentation with Large Language Model

GSVA: Generalized Segmentation via Multimodal Large Language Models

General Object Foundation Model for Images and Videos at Scale

PixelLM: Pixel Reasoning with Large Multimodal Model

Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives

PerceptionGPT: Effectively Fusing Visual Perception Into LLM

u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model

NExT-Chat: An LMM for Chat, Detection and Segmentation

CogVLM: Visual Expert for Pretrained Language Models

GLaMM: Pixel Grounding Large Multimodal Model

Ferret: Refer and Ground Anything Anywhere at Any Granularity

Improved Baselines with Visual Instruction Tuning

DreamLLM: Synergistic Multimodal Comprehension and Creation

Textbooks Are All You Need II: phi-1.5 technical report

LISA: Reasoning Segmentation via Large Language Model

Llama 2: Open Foundation and Fine-Tuned Chat Models

Generative Pretraining in Multimodality

Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic

Kosmos-2: Grounding Multimodal Large Language Models to the World

OBELISC: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents

GRES: Generalized Referring Expression Segmentation

InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

Visual Instruction Tuning

Segment Everything Everywhere All at Once

SegGPT: Segmenting Everything In Context

Segment Anything

Universal Instance Perception as Object Discovery and Retrieval

Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models

Side Adapter Network for Open-Vocabulary Semantic Segmentation

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

PACO: Parts and Attributes of Common Objects

Generalized Decoding for Pixel, Image, and Language

Images Speak in Images: A Generalist Painter for In-Context Visual Learning

OneFormer: One Transformer to Rule Universal Image Segmentation

Open-Vocabulary Universal Image Segmentation with MaskCLIP

XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model

Large-scale Video Panoptic Segmentation in the Wild: A Benchmark

Flamingo: a Visual Language Model for Few-Shot Learning

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model

Masked-attention Mask Transformer for Universal Image Segmentation

Vision-Language Transformer and Query Generation for Referring Segmentation

Learning Transferable Visual Models From Natural Language Supervision

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

Language Models are Few-Shot Learners

Objects365: A Large-Scale, High-Quality Dataset for Object Detection

LVIS: A Dataset for Large Vocabulary Instance Segmentation

Expression

MAttNet: Modular Attention Network for Referring Expression Comprehension

Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations

The 2017 DAVIS Challenge on Video Object Segmentation

COCO-Stuff: Thing and Stuff Classes in Context

Semantic Understanding of Scenes Through the ADE20K Dataset

Modeling Context in Referring Expressions

The Cityscapes Dataset for Semantic Urban Scene Understanding

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

The Role of Context for Object Detection and Semantic Segmentation in the Wild

Detect What You Can: Detecting and Representing Objects Using Holistic Models and Body Parts

Microsoft COCO: Common Objects in Context

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

The PASCAL visual object classes challenge 2006 (VOC2006) results

Gemini: a family of highly capable multimodal models

OpenAI

Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names