computer-vision-1

Zero-Shot Video Question Answer

3260 papers • 126 benchmarks • 313 datasets

This task present the results of Zeroshot Question Answer results on TGIF-QA dataset for LLM powered Video Conversational Models.

(Image credit: Papersgraph)

Benchmarks

These leaderboards are used to track progress in zero-shot-video-question-answer-2

Trend

Dataset

Best Model

Actions

MSRVTT-QA

ActivityNet-QA

MSVD-QA

Libraries

i

Use these libraries to find zero-shot-video-question-answer-2 models and implementations

pku-yuangroup/video-bench

2 papers 110

Datasets

Subtasks

No subtasks available.

Most implemented papers

Flamingo: a Visual Language Model for Few-Shot Learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, A. Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, O. Vinyals, Andrew Zisserman, K. Simonyan•Thu Apr 28 2022

This work introduces Flamingo, a family of Visual Language Models (VLM) with this ability to bridge powerful pretrained vision-only and language-only models, handle sequences of arbitrarily interleaved visual and textual data, and seamlessly ingest images or videos as inputs.

Content

EgoSchema (fullset)

NExT-QA

TGIF-QA

EgoSchema (subset)

TVQA

IntentQA

NExT-GQA

STAR Benchmark

MVBench

CinePile: A Long Video Question Answering Dataset and Benchmark

Ziyang412/VideoTree

2 papers 53

MSVD-QA

4615 0

Video-LLaVA: Learning United Visual Representation by Alignment Before Projection

Bin Lin, Munan Ning, Bin Zhu, Peng Jin, Yang Ye, Li Yuan•Wed Nov 15 2023

This work unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM, and establishes a simple but robust LVLM baseline, Video-LLaVA, which learns from a mixed dataset of images and videos, mutually enhancing each other.

1230 0

Paper Graph

Mistral 7B

A. Mensch, Guillaume Lample, Thibaut Lavril, Diego de Las Casas, M. Lachaux, Thomas Wang, Teven Le Scao, Timothée Lacroix, Albert Qiaochu Jiang, Alexandre Sablayrolles, Chris Bamford, Devendra Singh Chaplot, Florian Bressand, Gianna Lengyel, Lucile Saulnier, Lélio Renard Lavaud, Pierre Stock, William El Sayed•Mon Oct 09 2023

This work introduces Mistral 7B v0.1, a 7-billion-parameter language model engineered for superior performance and efficiency, which leverages grouped-query attention (GQA) for faster inference, coupled with sliding window attention (SWA) to effectively handle sequences of arbitrary length with a reduced inference cost.

3033 0

Paper Graph

LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model

Hongsheng Li, Conghui He, Pan Lu, Renrui Zhang, Peng Gao, W. Zhang, Xiangyu Yue, Y. Qiao, Jiaming Han, Aojun Zhou, Ziyi Lin, Shijie Geng•Thu Apr 27 2023

This work augments LLaMA-Adapter by unlocking more learnable parameters and proposes an early fusion strategy to feed visual tokens only into the early LLM layers, contributing to better visual knowledge incorporation and achieves strong multi-modal reasoning with only a small-scale image-text and instruction dataset.

714 0

Paper Graph

MVBench: A Comprehensive Multi-modal Video Understanding Benchmark

Yali Wang, Kunchang Li, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, Yu Qiao•Mon Nov 27 2023

A comprehensive Multi-modal Video understanding Benchmark, namely MVBench, which covers 20 chal-lenging video tasks that cannot be effectively solved with a single frame, and develops a robust video MLLM baseline, i.e., VideoChat2, by progressive multi-modal training with di-verse instruction-tuning data.

872 0

Paper Graph

TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering

Gunhee Kim, Yale Song, Y. Jang, Youngjae Yu, Youngjin Kim•Thu Apr 13 2017

This paper proposes three new tasks designed specifically for video VQA, which require spatio-temporal reasoning from videos to answer questions correctly and introduces a new large-scale dataset for videoVQA named TGIF-QA that extends existing VQ a work with its new tasks.

634 0

Paper Graph

VILA: On Pre-training for Visual Language Models

Huizi Mao, Pavlo Molchanov, Jan Kautz, Wei Ping, M. Shoeybi, Hongxu Yin, Ji Lin, Yao Lu, Andrew Tao, Song Han•Mon Dec 11 2023

This work examines the design options for VLM pre-training by augmenting LLM towards VLM through step-by-step controllable comparisons, and builds VILA, a Visual Language model family that consistently outperforms the state-of-the-art models, e.g., LLaVA-1.5, across main benchmarks without bells and whistles.

695 0

Paper Graph

MVB: A Large-Scale Dataset for Baggage Re-Identification and Merged Siamese Networks

Zhulin Zhang, Dong Li, Jinhua Wu, Yunda Sun, Li Zhang•Thu Jul 25 2019

MVB (Multi View Baggage) is the first publicly released large-scale dataset that contains 4519 baggage identities and 22660 annotated baggage images as well as its surface material labels and has remarkable inter-class similarity and intra-class dissimilarity.

13 0

Paper Graph

Zero-Shot Video Question Answering via Frozen Bidirectional Language Models

Antoine Miech, I. Laptev, C. Schmid, Josef Sivic, Antoine Yang•Wed Jun 15 2022

This work builds on frozen bidirectional language models (BiLM) and shows that such an approach provides a stronger and cheaper alternative for zero-shot VideoQA and demonstrates competitive performance in the few-shot and fully-supervised setting.

277 0

Paper Graph

InternVideo: General Video Foundation Models via Generative and Discriminative Learning

Limin Wang, Yi Wang, Jiashuo Yu, Yali Wang, Yu Qiao, Junting Pan, Kunchang Li, Yinan He, Yizhuo Li, Zun Wang, Jilan Xu, Guo Chen, Hongjie Zhang, Bingkun Huang, Zhiyu Zhao, Yi Liu, Sen Xing•Mon Dec 05 2022

Without bells and whistles, InternVideo achieves state-of-the-art performance on 39 video datasets from extensive tasks including video action recognition/detection, video-language alignment, and open-world video applications.

457 0

Paper Graph

Adding a benchmark result helps the community track progress.