Zero-Shot and Few-Shot Video Question Answering with Multi-Modal Prompts (2023-09-27T00:00:00.000000Z)

TL;DR

A parameter-efficient method is introduced, combining multimodal prompt learning and a transformer-based mapping network, while keeping the pretrained models frozen to address challenges such as overfitting, catastrophic forgetting, and the cross-modal gap between vision and language.

Abstract

Recent vision-language models are driven by large-scale pretrained models. However, adapting pretrained models on limited data presents challenges such as overfitting, catastrophic forgetting, and the cross-modal gap between vision and language. We introduce a parameter-efficient method to address these challenges, combining multimodal prompt learning and a transformer-based mapping network, while keeping the pretrained models frozen. Our experiments on several video question answering benchmarks demonstrate the superiority of our approach in terms of performance and parameter efficiency on both zero-shot and few-shot settings. Our code is available at https://engindeniz.github.io/vitis.

Authors

Yannis Avrithis

4 papers

Deniz Engin

2 papers

TL;DR

Abstract

Authors

References64 items

Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting

AutoAD: Movie Description in Context

Multimodal Prompting with Missing Modalities for Visual Recognition

VindLU: A Recipe for Effective Video-and-Language Pretraining

Fine-tuned CLIP Models are Efficient Video Learners

MaPLe: Multi-modal Prompt Learning

Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models

An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling

Clover: Towards A Unified Video-Language Alignment and Fusion Model

Zero-Shot Video Question Answering via Frozen Bidirectional Language Models

LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling

Learning to Answer Visual Questions From Web Videos

Flamingo: a Visual Language Model for Few-Shot Learning

Prompt-free and Efficient Few-shot Learning with Language Models

Exploring Visual Prompts for Adapting Large-Scale Models

Visual Prompt Tuning

All in One: Exploring Unified Video-Language Pre-Training

Conditional Prompt Learning for Vision-Language Models

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound

Align and Prompt: Video-and-Language Pre-training with Entity Prompts

VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning

Prompting Visual-Language Models for Efficient Video Understanding

Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks

VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling

ClipCap: CLIP Prefix for Image Captioning

Learning to Prompt for Vision-Language Models

Video Swin Transformer

MERLOT: Multimodal Neural Script Knowledge Models

The Power of Scale for Parameter-Efficient Prompt Tuning

Learning How to Ask: Querying LMs with Mixtures of Soft Prompts

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

GPT Understands, Too

Perceiver: General Perception with Iterative Attention

Learning Transferable Visual Models From Natural Language Supervision

Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling

Just Ask: Learning to Answer Questions from Millions of Narrated Videos

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Language Models are Few-Shot Learners

Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training

KnowIT VQA: Answering Knowledge-Based Questions about Videos

RoBERTa: A Robustly Optimized BERT Pretraining Approach

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering

VideoBERT: A Joint Model for Video and Language Representation Learning

Parameter-Efficient Transfer Learning for NLP

TVQA: Localized, Compositional Video Question Answering

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Video Question Answering via Gradually Refined Attention over Appearance and Motion

TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

MSR-VTT: A Large Video Description Dataset for Bridging Video and Language

MovieQA: Understanding Stories in Movies through Question-Answering

ActivityNet: A large-scale video benchmark for human activity understanding

Adam: A Method for Stochastic Optimization

Collecting Highly Parallel Data for Paraphrase Evaluation

P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks

Prefix-Tuning: Optimizing Continuous Prompts for Generation

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

We experimentally demonstrate the strong performance of ViTiS on multiple VideoQA datasets in both zero-shot and few-shot settings

applicable license agreement with IEEE. Restrictions apply

Field of Study

Journal Information

Name

Page