OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding (2024-06-27T00:00:00.000000Z)

TL;DR

This paper uses a universal segmentation method as the visual encoder, integrating image information, perception priors, and visual prompts into visual tokens provided to the LLM, and proposes perception prior embedding to better integrate perception priors with image features.

Abstract

Current universal segmentation methods demonstrate strong capabilities in pixel-level image and video understanding. However, they lack reasoning abilities and cannot be controlled via text instructions. In contrast, large vision-language multimodal models exhibit powerful vision-based conversation and reasoning capabilities but lack pixel-level understanding and have difficulty accepting visual prompts for flexible user interaction. This paper proposes OMG-LLaVA, a new and elegant framework combining powerful pixel-level vision understanding with reasoning abilities. It can accept various visual and text prompts for flexible user interaction. Specifically, we use a universal segmentation method as the visual encoder, integrating image information, perception priors, and visual prompts into visual tokens provided to the LLM. The LLM is responsible for understanding the user's text instructions and providing text responses and pixel-level segmentation results based on the visual information. We propose perception prior embedding to better integrate perception priors with image features. OMG-LLaVA achieves image-level, object-level, and pixel-level reasoning and understanding in a single model, matching or surpassing the performance of specialized methods on multiple benchmarks. Rather than using LLM to connect each specialist, our work aims at end-to-end training on one encoder, one decoder, and one LLM. The code and model have been released for further research.

Authors

Shengqiong Wu

4 papers

Chen Change Loy

4 papers

Haobo Yuan

4 papers

TL;DR

Abstract

Authors

References139 items

Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing

Qwen2 Technical Report

Towards Semantic Equivalence of Tokenization in Multimodal LLM

Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model

Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition

How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites

Enhancing Video-Language Representations With Structural Spatio-Temporal Alignment

Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models

LaSagnA: Language-based Segmentation Assistant for Complex Queries

LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning

InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD

Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want

DVIS-DAQ: Improving Video Segmentation via Dynamic Anchor Queries

Are We on the Right Way for Evaluating Large Vision-Language Models?

Mini-Gemini: Mining the Potential of Multi-Modality Vision Language Models

InternLM2 Technical Report

PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model

LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images

Multi-Modal Instruction Tuned LLMs with Fine-Grained Visual Perception

Groundhog Grounding Large Language Models to Holistic Segmentation

Generalizable Entity Grounding via Assistance of Large Language Model

InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model

OMG-Seg: Is One Model Good Enough for all Segmentation?

Towards Language-Driven Video Inpainting via Multimodal Large Language Models

Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively

An Improved Baseline for Reasoning Segmentation with Large Language Model

Generative Multimodal Models are In-Context Learners

Osprey: Pixel Understanding with Visual Instruction Tuning

GSVA: Generalized Segmentation via Multimodal Large Language Models

Tokenize Anything via Prompting

See, Say, and Segment: Teaching LMMs to Overcome False Premises

Skeleton-in-Context: Unified Skeleton Sequence Modeling with In-Context Learning

PixelLM: Pixel Reasoning with Large Multimodal Model

ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts

LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models

Visual in-Context Prompting

ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

Video-LLaVA: Learning United Visual Representation by Alignment Before Projection

LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents

NExT-Chat: An LMM for Chat, Detection and Segmentation

GLaMM: Pixel Grounding Large Multimodal Model

LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editing

Ferret: Refer and Ground Anything Anywhere at Any Granularity

Improved Baselines with Visual Instruction Tuning

InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition

NExT-GPT: Any-to-Any Multimodal LLM

Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models

Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions

LISA: Reasoning Segmentation via Large Language Model

SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension

Llama 2: Open Foundation and Fine-Tuned Chat Models

MMBench: Is Your Multi-modal Model an All-around Player?

GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest

ReMaX: Relaxing for Better Training on Efficient Panoptic Segmentation

Towards Open Vocabulary Learning: A Survey

Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic

Kosmos-2: Grounding Multimodal Large Language Models to the World

MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

Explore In-Context Learning for 3D Point Cloud Understanding

Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model

GRES: Generalized Referring Expression Segmentation

Contextual Object Detection with Multimodal Large Language Models

VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks

Evaluating Object Hallucination in Large Vision-Language Models

Segment and Track Anything

In-Context Learning Unlocked for Diffusion Models

MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

Transformer-Based Visual Segmentation: A Survey

Visual Instruction Tuning

SegGPT: Segmenting Everything In Context

Segment Anything

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

Universal Instance Perception as Object Discovery and Retrieval