How Much Can CLIP Benefit Vision-and-Language Tasks?

Published in

International Conference on Learning Representa...(2021)

External Links:

Generate Graph

TL;DR

It is shown that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data, such as BottomUp-TopDown, and achieves competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering, Visual Entailment, and V & L Navigation tasks.

Abstract

Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world. However, it has been observed that large-scale pretraining usually can result in better generalization performance, e.g., CLIP (Contrastive Language-Image Pre-training), trained on a massive amount of image-caption pairs, has shown a strong zero-shot capability on various vision tasks. To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in two typical scenarios: 1) plugging CLIP into task-specific fine-tuning; 2) combining CLIP with V&L pre-training and transferring to downstream tasks. We show that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data, such as BottomUp-TopDown. We achieve competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering, Visual Entailment, and V&L Navigation tasks. We release our code at https://github.com/clip-vil/CLIP-ViL.

Authors

Z. Yao

4 papers

Sheng Shen

9 papers

K. Keutzer

18 papers

References89 items

Reward Shaping Study for Sub-Goal Based Robotic Tasks

Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions

VinVL: Revisiting Visual Representations in Vision-Language Models

Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information

Learning Transferable Visual Models From Natural Language Supervision

How Much Can CLIP Benefit Vision-and-Language Tasks?

Published in

International Conference on Learning Representa...(2021)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Z. Yao

4 papers

Sheng Shen

9 papers

K. Keutzer

18 papers

References89 items

Reward Shaping Study for Sub-Goal Based Robotic Tasks

Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions

VinVL: Revisiting Visual Representations in Vision-Language Models

Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information

Learning Transferable Visual Models From Natural Language Supervision

Mohit Bansal

47 papers

Anna Rohrbach

16 papers

Kai-Wei Chang

20 papers

Liunian Harold Li

3 papers

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Language and Visual Entity Relationship Graph for Agent Navigation

Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding

Learning Visual Representations with Caption Annotations

VirTex: Learning Visual Representations from Textual Annotations

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments

Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers

A Better Variant of Self-Critical Sequence Training

Stanza: A Python Natural Language Processing Toolkit for Many Human Languages

Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training

In Defense of Grid Features for Visual Question Answering

Vision-Language Navigation With Self-Supervised Auxiliary Reasoning Tasks

Objects365: A Large-Scale, High-Quality Dataset for Object Detection

UNITER: UNiversal Image-TExt Representation Learning

Unified Vision-Language Pre-Training for Image Captioning and VQA

Robust Navigation with Language Pretraining and Stochastic Sampling

Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Attention on Attention for Image Captioning

Transferable Representation Learning in Vision-and-Language Navigation

VisualBERT: A Simple and Performant Baseline for Vision and Language

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Chasing Ghosts: Instruction Following as Bayesian State Tracking

Deep Modular Co-Attention Networks for Visual Question Answering

Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation

REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments

Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout

Tactical Rewind: Self-Correction via Backtracking in Vision-And-Language Navigation

The Regretful Agent: Heuristic-Aided Navigation Through Progress Estimation

GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

Visual Entailment: A Novel Task for Fine-Grained Image Understanding

Self-Monitoring Navigation Agent via Auxiliary Progress Estimation

Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering

TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Pythia v0.1: the Winning Entry to the VQA Challenge 2018

Speaker-Follower Models for Vision-and-Language Navigation

Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation

Discriminability Objective for Training Descriptive Captions

Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments

Decoupled Weight Decay Regularization

Matterport3D: Learning from RGB-D Data in Indoor Environments

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Attention is All you Need

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Self-Critical Sequence Training for Image Captioning

Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

SPICE: Semantic Propositional Image Caption Evaluation

Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Learning Deep Features for Discriminative Localization

Deep Residual Learning for Image Recognition

Learning Deep Structure-Preserving Image-Text Embeddings

Visual7W: Grounded Question Answering in Images

Stacked Attention Networks for Image Question Answering

VQA: Visual Question Answering

Microsoft COCO Captions: Data Collection and Evaluation Server

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Deep visual-semantic alignments for generating image descriptions

CIDEr: Consensus-based image description evaluation

ImageNet Large Scale Visual Recognition Challenge

Microsoft COCO: Common Objects in Context

METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments

Bleu: a Method for Automatic Evaluation of Machine Translation

Long Short-Term Memory

publicly available implementation 10 . During the self-critical stage, we sample 5 captions for each image as in Luo (

CLIP visual module using implementation of Faster R-CNN in Detectron2 8 . For training VQA models, we use hyperparameters of the open-source implementation 9 from

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Implementation Details We apply our model to two vision-and-language navigation datasets: Room-to-Room

The agent model (i.e., another LSTM) then at-tends to the visual features and the language representations to predict the actions

metric is optimized using REINFORCE algorithm (Williams, 1992).

network and fuse the information with a multi-modal factorized bilinear pooling network

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Since RxR dataset contains instructions longer than R2R, we change the maximum input length to 160 11

A Appendix A.1 Visual Question Answering Model Architecture

A.2 Image Captioning Implementation Details For training, we follow the ‘long epoch’

pooling. Implementation Details We follow (Jiang et al. 2020) to resize all input images to have a maximum

shorter side of 600 pixels (longest 1000) when keeping the aspect ratio ﬁxed

The open images dataset v 4 : Unified image classification , object detection , and visual relationship detection at scale

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

International Conference on Learning Representations

Type

conference

URL

https://iclr.cc/

Alternate Names

Int Conf Learn Represent
ICLR