VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021-09-28T00:00:00.000000Z)

TL;DR

VideoCLIP is presented, a contrastive approach to pre-train a unified model for zero-shot video and text understanding, without using any labels on downstream tasks, revealing state-of-the-art performance, surpassing prior work, and in some cases even outperforming supervised approaches.

Abstract

We present VideoCLIP, a contrastive approach to pre-train a unified model for zero-shot video and text understanding, without using any labels on downstream tasks. VideoCLIP trains a transformer for video and text by contrasting temporally overlapping positive video-text pairs with hard negatives from nearest neighbor retrieval. Our experiments on a diverse series of downstream tasks, including sequence-level text-video retrieval, VideoQA, token-level action localization, and action segmentation reveal state-of-the-art performance, surpassing prior work, and in some cases even outperforming supervised approaches. Code is made available at https://github.com/pytorch/fairseq/examples/MMPT.

Authors

Po-Yao (Bernie) Huang

5 papers

Dmytro Okhonko

3 papers

Gargi Ghosh

3 papers

TL;DR

Abstract

Authors

References68 items

VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding

CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval

Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models

Learning Transferable Visual Models From Natural Language Supervision

A Straightforward Framework For Video Retrieval Using CLIP

Zero-Shot Text-to-Image Generation

Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling

COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning

Support-set bottlenecks for video-text representation learning

Multi-modal Transformer for Video Retrieval

Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering

Self-Supervised MultiModal Versatile Networks

Pre-training via Paraphrasing

Video Understanding as Machine Translation

ActBERT: Learning Global-Local Video-Text Representations

Language Models are Few-Shot Learners

Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training

Dense Passage Retrieval for Open-Domain Question Answering

UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation

A Simple Framework for Contrastive Learning of Visual Representations

REALM: Retrieval-Augmented Language Model Pre-Training

End-to-End Learning of Visual Representations From Uncurated Instructional Videos

Momentum Contrast for Unsupervised Visual Representation Learning

Generalization through Memorization: Nearest Neighbor Language Models

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

UNITER: UNiversal Image-TExt Representation Learning

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Use What You Have: Video retrieval using representations from collaborative experts

RoBERTa: A Robustly Optimized BERT Pretraining Approach

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Contrastive Bidirectional Transformer for Temporal Representation Learning

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

VideoBERT: A Joint Model for Video and Language Representation Learning

Cross-Task Weakly Supervised Learning From Instructional Videos

COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis

SlowFast Networks for Video Recognition

Cross-Modal and Hierarchical Modeling of Video and Text

A Joint Sequence Fusion Model for Video Question Answering and Retrieval

Representation Learning with Contrastive Predictive Coding

NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning

End-to-End Dense Video Captioning with Masked Transformer

Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment

Deep Contextualized Word Representations

Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification

Localizing Moments in Video with Natural Language

Attention is All you Need

Towards Automatic Learning of Procedures From Web Instructional Videos

Temporal Tessellation: A Unified Approach for Video Analysis

End-to-End Concept Word Detection for Video Captioning, Retrieval, and Question Answering

Video Captioning and Retrieval Models with Semantic Attention

Learning Language-Visual Embedding for Movie Understanding with Natural-Language

Enriching Word Vectors with Subword Information

Gaussian Error Linear Units (GELUs)

MSR-VTT: A Large Video Description Dataset for Bridging Video and Language

Unsupervised Learning from Narrated Instruction Videos

Associating neural word embeddings with deep image representations using Fisher Vectors

Adam: A Method for Stochastic Optimization

Translating Videos to Natural Language Using Deep Recurrent Neural Networks

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Very Deep Convolutional Networks for Large-Scale Image Recognition

Distributed Representations of Words and Phrases and their Compositionality

Toward Open Set Recognition

Out-of-Domain Utterance Detection Using Classification Confidences of Multiple Topics

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Improving Language Understanding by Generative Pre-Training

Hadamard product for low-rank bilinear pooling

Field of Study

Venue Information

Name

Type

URL

Alternate Names