Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023-11-14T00:00:00.000000Z)

TL;DR

This work introduces Chat-UniVi, a Unified Vision-language model capable of comprehending and engaging in conver-sations involving images and videos through a unified visual representation that outperforms even existing methods exclusively de-signed for either images or videos.

Abstract

Large language models have demonstrated impressive universal capabilities across a wide range of open-ended tasks and have extended their utility to encompass multi-modal conversations. However, existing methods encounter challenges in effectively handling both image and video understanding, particularly with limited visual tokens. In this work, we introduce Chat-UniVi, a Unified Vision-language model capable of comprehending and engaging in conver-sations involving images and videos through a unified visual representation. Specifically, we employ a set of dynamic visual tokens to uniformly represent images and videos. This representation framework empowers the model to ef-ficiently utilize a limited number of visual tokens to simul-taneously capture the spatial details necessary for images and the comprehensive temporal relationship required for videos. Moreover, we leverage a multi-scale representation, enabling the model to perceive both high-level seman-tic concepts and low-level visual details. Notably, Chat-UniVi is trained on a mixed dataset containing both images and videos, allowing direct application to tasks involving both mediums without requiring any modifications. Exten-sive experimental results demonstrate that Chat- UniVi con-sistently outperforms even existing methods exclusively de-signed for either images or videos. Code is available at https://github.com/PKu-Yuan Group/Chat-UniVi.

Authors

Ryuichi Takanobu

3 papers

Peng Jin

4 papers

Li Yuan

2 papers

TL;DR

Abstract

Authors

References86 items

LLMBind: A Unified Modality-Task Integration Framework

SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models

MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding

Video-LLaVA: Learning United Visual Representation by Alignment Before Projection

TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding

MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning

Improved Baselines with Visual Instruction Tuning

LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment

MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens

NExT-GPT: Any-to-Any Multimodal LLM

OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models

Llama 2: Open Foundation and Fine-Tuned Chat Models

SCITUNE: Aligning Large Language Models with Scientific Multimodal Instructions

LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding

Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic

Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning

Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models

Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models

Text-Video Retrieval with Disentangled Conceptualization and Set-to-Set Alignment

Evaluating Object Hallucination in Large Vision-Language Models

InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

VideoChat: chat-centric video understanding

MultiModal-GPT: A Vision and Language Model for Dialogue with Humans

X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages

Otter: A Multi-Modal Model With In-Context Instruction Tuning

LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model

mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality

MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

Visual Instruction Tuning

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

EVA-CLIP: Improved Training Techniques for CLIP at Scale

Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learning

MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action

DiffusionRet: Generative Text-Video Retrieval with Diffusion Model

GPT-4 Technical Report

ViperGPT: Visual Inference via Python Execution for Reasoning

Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models

Image as Set of Points

LLaMA: Open and Efficient Foundation Language Models

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

Token Merging: Your ViT But Faster

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

OmniVL: One Foundation Model for Image-Language and Video-Language Tasks

Zero-Shot Video Question Answering via Frozen Bidirectional Language Models

OPT: Open Pre-trained Transformer Language Models

Flamingo: a Visual Language Model for Few-Shot Learning

Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformer

PaLM: Scaling Language Modeling with Pathways

Training language models to follow instructions with human feedback

GroupViT: Semantic Segmentation Emerges from Text Supervision

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

Scaling Language Models: Methods, Analysis & Insights from Training Gopher

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

LoRA: Low-Rank Adaptation of Large Language Models

DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

Learning Transferable Visual Models From Natural Language Supervision

Calibrate Before Use: Improving Few-Shot Performance of Language Models

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Language Models are Few-Shot Learners

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering

GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

Decoupled Weight Decay Regularization

Video Question Answering via Gradually Refined Attention over Appearance and Motion

Attention is All you Need

TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering

Study on density peaks clustering based on k-nearest neighbors and principal component analysis

ActivityNet: A large-scale video benchmark for human activity understanding

VQA: Visual Question Answering

Microsoft COCO Captions: Data Collection and Evaluation Server