Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models (2023-06-08T00:00:00.000000Z)

TL;DR

A multimodal model that merges a video-adapted visual encoder with an LLM, capable of understanding and generating detailed conversations about videos, and a quantitative evaluation framework for video-based dialogue models to objectively analyze the strengths and weaknesses of video-based dialogue models.

Abstract

Conversation agents fueled by Large Language Models (LLMs) are providing a new way to interact with visual data. While there have been initial attempts for image-based conversation models, this work addresses the under-explored field of \emph{video-based conversation} by introducing Video-ChatGPT. It is a multimodal model that merges a video-adapted visual encoder with an LLM. The resulting model is capable of understanding and generating detailed conversations about videos. We introduce a new dataset of 100,000 video-instruction pairs used to train Video-ChatGPT acquired via manual and semi-automated pipeline that is easily scalable and robust to label noise. We also develop a quantitative evaluation framework for video-based dialogue models to objectively analyze the strengths and weaknesses of video-based dialogue models. Code: https://github.com/mbzuai-oryx/Video-ChatGPT.

Authors

F. Khan

29 papers

Salman H. Khan

6 papers

Muhammad Maaz

6 papers

TL;DR

Abstract

Authors

References40 items

Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

PandaGPT: One Model To Instruction-Follow Them All

InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

VideoChat: chat-centric video understanding

LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model

MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

Tag2Text: Guiding Vision-Language Model via Image Tagging

LLaMA: Open and Efficient Foundation Language Models

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Fine-tuned CLIP Models are Efficient Video Learners

InternVideo: General Video Foundation Models via Generative and Discriminative Learning

GRiT: A Generative Region-to-text Transformer for Object Understanding

Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP

Expanding Language-Image Pretrained Models for General Video Recognition

Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection

Zero-Shot Video Question Answering via Frozen Bidirectional Language Models

OPT: Open Pre-trained Transformer Language Models

Flamingo: a Visual Language Model for Few-Shot Learning

Language-Grounded Indoor 3D Semantic Segmentation in the Wild

Training language models to follow instructions with human feedback

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

Class-Agnostic Object Detection with Multi-modal Transformer

ActionCLIP: A New Paradigm for Video Action Recognition

Towards General Purpose Vision Systems: An End-to-End Task-Agnostic Vision-Language Architecture

Probabilistic two-stage detection

Learning Transferable Visual Models From Natural Language Supervision

Language Models are Few-Shot Learners

ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering

Video Question Answering via Gradually Refined Attention over Appearance and Motion

TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering

ActivityNet: A large-scale video benchmark for human activity understanding

Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023

2023. Visual instruction tuning

iv) Temporal Understanding: We examine the model’s grasp of the temporal sequence of events in the video when answering questions

OpenAI

Katna : Tool for automating video keyframe extraction , video compression , image autocrop and smart image resize tasks

Stanford alpaca: An instruction-following llama model

Chatgpt. Large Language Model for human style conversation

: Codebase for moss project. An open-sourced plugin-augmented conversational language model

iii) Contextual Understanding: We assess the model’s understanding of the video’s context, checking if its responses aligns with the overall context of the video content

Field of Study

Venue Information

Name

Type

URL

Alternate Names