End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019-12-13T00:00:00.000000Z)

TL;DR

This work proposes a new learning approach, MIL-NCE, capable of addressing mis- alignments inherent in narrated videos and outperforms all published self-supervised approaches for these tasks as well as several fully supervised baselines.

Abstract

Annotating videos is cumbersome, expensive and not scalable. Yet, many strong video models still rely on manually annotated data. With the recent introduction of the HowTo100M dataset, narrated videos now offer the possibility of learning video representations without manual supervision. In this work we propose a new learning approach, MIL-NCE, capable of addressing mis- alignments inherent in narrated videos. With this approach we are able to learn strong video representations from scratch, without the need for any manual annotation. We evaluate our representations on a wide range of four downstream tasks over eight datasets: action recognition (HMDB-51, UCF-101, Kinetics-700), text-to- video retrieval (YouCook2, MSR-VTT), action localization (YouTube-8M Segments, CrossTask) and action segmentation (COIN). Our method outperforms all published self-supervised approaches for these tasks as well as several fully supervised baselines.

Authors

Jean-Baptiste Alayrac

8 papers

Antoine Miech

12 papers

Andrew Zisserman

62 papers

TL;DR

Abstract

Authors

References129 items

Learning Video Representations using Contrastive Bidirectional Transformer

Video Representation Learning by Dense Predictive Coding

Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings

A Short Note on the Kinetics-700 Human Action Dataset

Contrastive Multiview Coding

Contrastive Bidirectional Transformer for Temporal Representation Learning

Grounding Object Detections With Transcriptions

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

Mining YouTube - A dataset for learning fine-grained action concepts from webly supervised video data

Self-Supervised Spatiotemporal Learning via Video Clip Order Prediction

Multimodal Abstractive Summarization for How2 Videos

Toward Self-Supervised Object Detection in Unlabeled Videos

Learning to Detect and Retrieve Objects From Unlabeled Videos

Data-Efficient Image Recognition with Contrastive Predictive Coding

Unsupervised Pre-Training of Image Features on Non-Curated Data

Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition

Temporal Cycle-Consistency Learning

Self-Supervised Spatio-Temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics

VideoBERT: A Joint Model for Video and Language Representation Learning

Cross-Task Weakly Supervised Learning From Instructional Videos

COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis

Self-supervised Spatiotemporal Feature Learning by Video Geometric Transformations

Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction.

Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles

How2: A Large-scale Dataset for Multimodal Language Understanding

Dual Encoding for Zero-Example Video Retrieval

Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval

Representation Learning with Contrastive Predictive Coding

A flexible model for training action localization with varying levels of supervision

Tracking Emerges by Colorizing Videos

Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval

Geometry Guided Convolutional Neural Networks for Self-Supervised Video Representation Learning

Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization

Learning a Text-Video Embedding from Incomplete and Heterogeneous Data

Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input

Attention-based Deep Multiple Instance Learning

Real-World Anomaly Detection in Surveillance Videos

Objects that Sound

Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification

Localizing Moments in Video with Natural Language

Unsupervised Representation Learning by Sorting Sequences

Learning from Video and Text via Large-Scale Discriminative Clustering

Enhancing Video Summarization via Vision-Language Embedding

Discover and Learn New Objects from Documentaries

Sampling Matters in Deep Embedding Learning

Attention is All you Need

Look, Listen and Learn

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

Learning Two-Branch Neural Networks for Image-Text Matching Tasks

Towards Automatic Learning of Procedures From Web Instructional Videos

Video Captioning and Retrieval Models with Semantic Attention

YouTube-8M: A Large-Scale Video Classification Benchmark

Instance-Level Video Segmentation from Object Tracks

MSR-VTT: A Large Video Description Dataset for Bridging Video and Language

Minding the Gaps for Block Frank-Wolfe Optimization of Structured SVMs

Towards Weakly-Supervised Action Localization

Movie Description

Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification

Exploring the Limits of Language Modeling

Deep Residual Learning for Image Recognition

NetVLAD: CNN Architecture for Weakly Supervised Place Recognition

Learning Deep Structure-Preserving Image-Text Embeddings

An Improved Non-monotonic Transition System for Dependency Parsing

Unsupervised Learning from Narrated Instruction Videos

Associating neural word embeddings with deep image representations using Fisher Vectors

Is object localization for free? - Weakly-supervised learning with convolutional neural networks

ActivityNet: A large-scale video benchmark for human activity understanding

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Weakly-Supervised Alignment of Video with Text

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

Action recognition with trajectory-pooled deep-convolutional descriptors

Jointly Modeling Embedding and Translation to Bridge Video and Language

Unsupervised Learning of Visual Representations Using Videos

Anticipating Visual Representations from Unlabeled Video

What’s Cookin’? Interpreting Cooking Videos using Text, Speech and Vision

Jointly Modeling Deep Video and Compositional Text to Bridge Vision and Language in a Unified Framework