VTG-GPT: Tuning-Free Zero-Shot Video Temporal Grounding with GPT
TL
TL;DR
VTG-GPT is proposed, a GPT-based method for zero-shot VTG without training or fine-tuning that significantly outperforms SOTA methods in zero-shot settings and surpasses unsupervised approaches and achieves competitive performance comparable to supervised methods.
Abstract
Video temporal grounding (VTG) aims to locate specific temporal segments from an untrimmed video based on a linguistic query. Most existing VTG models are trained on extensive annotated video-text pairs, a process that not only introduces human biases from the queries but also incurs significant computational costs. To tackle these challenges, we propose VTG-GPT, a GPT-based method for zero-shot VTG without training or fine-tuning. To reduce prejudice in the original query, we employ Baichuan2 to generate debiased queries. To lessen redundant information in videos, we apply MiniGPT-v2 to transform visual content into more precise captions. Finally, we devise the proposal generator and post-processing to produce accurate segments from debiased queries and image captions. Extensive experiments demonstrate that VTG-GPT significantly outperforms SOTA methods in zero-shot settings and surpasses unsupervised approaches. More notably, it achieves competitive performance comparable to supervised methods. The code is available on GitHub.
References48 items
1
GPTSee: Enhancing Moment Retrieval and Highlight Detection via Description-Based Similarity Features
2
Dynamic Contrastive Learning with Pseudo-samples Intervention for Weakly Supervised Joint Video MR and HD
3
Query-Guided Refinement and Dynamic Spans Network for Video Highlight Detection and Temporal Grounding in Online Information Systems
4
MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning
5