How Much Can CLIP Benefit Vision-and-Language Tasks? (2021-07-13T00:00:00.000000Z)

TL;DR

It is shown that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data, such as BottomUp-TopDown, and achieves competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering, Visual Entailment, and V & L Navigation tasks.

Abstract

Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world. However, it has been observed that large-scale pretraining usually can result in better generalization performance, e.g., CLIP (Contrastive Language-Image Pre-training), trained on a massive amount of image-caption pairs, has shown a strong zero-shot capability on various vision tasks. To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in two typical scenarios: 1) plugging CLIP into task-specific fine-tuning; 2) combining CLIP with V&L pre-training and transferring to downstream tasks. We show that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data, such as BottomUp-TopDown. We achieve competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering, Visual Entailment, and V&L Navigation tasks. We release our code at https://github.com/clip-vil/CLIP-ViL.

Authors

Z. Yao

4 papers

Sheng Shen

9 papers

K. Keutzer

18 papers

TL;DR

Abstract

Authors

References89 items

Reward Shaping Study for Sub-Goal Based Robotic Tasks

Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions

VinVL: Revisiting Visual Representations in Vision-Language Models

Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information

Learning Transferable Visual Models From Natural Language Supervision

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Language and Visual Entity Relationship Graph for Agent Navigation

Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding

Learning Visual Representations with Caption Annotations

VirTex: Learning Visual Representations from Textual Annotations

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments

Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers

A Better Variant of Self-Critical Sequence Training

Stanza: A Python Natural Language Processing Toolkit for Many Human Languages

Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training

In Defense of Grid Features for Visual Question Answering

Vision-Language Navigation With Self-Supervised Auxiliary Reasoning Tasks

Objects365: A Large-Scale, High-Quality Dataset for Object Detection

UNITER: UNiversal Image-TExt Representation Learning

Unified Vision-Language Pre-Training for Image Captioning and VQA

Robust Navigation with Language Pretraining and Stochastic Sampling

Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Attention on Attention for Image Captioning

Transferable Representation Learning in Vision-and-Language Navigation

VisualBERT: A Simple and Performant Baseline for Vision and Language

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Chasing Ghosts: Instruction Following as Bayesian State Tracking

Deep Modular Co-Attention Networks for Visual Question Answering

Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation

REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments

Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout

Tactical Rewind: Self-Correction via Backtracking in Vision-And-Language Navigation

The Regretful Agent: Heuristic-Aided Navigation Through Progress Estimation

GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

Visual Entailment: A Novel Task for Fine-Grained Image Understanding

Self-Monitoring Navigation Agent via Auxiliary Progress Estimation

Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering

TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Pythia v0.1: the Winning Entry to the VQA Challenge 2018

Speaker-Follower Models for Vision-and-Language Navigation

Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation

Discriminability Objective for Training Descriptive Captions

Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments

Decoupled Weight Decay Regularization

Matterport3D: Learning from RGB-D Data in Indoor Environments

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Attention is All you Need

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Self-Critical Sequence Training for Image Captioning

Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

SPICE: Semantic Propositional Image Caption Evaluation

Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Learning Deep Features for Discriminative Localization

Deep Residual Learning for Image Recognition

Learning Deep Structure-Preserving Image-Text Embeddings

Visual7W: Grounded Question Answering in Images

Stacked Attention Networks for Image Question Answering

VQA: Visual Question Answering

Microsoft COCO Captions: Data Collection and Evaluation Server

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Deep visual-semantic alignments for generating image descriptions

CIDEr: Consensus-based image description evaluation

ImageNet Large Scale Visual Recognition Challenge

Microsoft COCO: Common Objects in Context

METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments

Bleu: a Method for Automatic Evaluation of Machine Translation

Long Short-Term Memory

publicly available implementation 10 . During the self-critical stage, we sample 5 captions for each image as in Luo (