Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts (2021-11-16T00:00:00.000000Z)

TL;DR

Experimental results show that X-VLM effectively leverages the learned multi-grained alignments to many downstream vision language tasks and consistently outperforms state-of-the-art methods.

Abstract

Most existing methods in vision language pre-training rely on object-centric features extracted through object detection and make fine-grained alignments between the extracted features and texts. It is challenging for these methods to learn relations among multiple objects. To this end, we propose a new method called X-VLM to perform `multi-grained vision language pre-training.' The key to learning multi-grained alignments is to locate visual concepts in the image given the associated texts, and in the meantime align the texts with the visual concepts, where the alignments are in multi-granularity. Experimental results show that X-VLM effectively leverages the learned multi-grained alignments to many downstream vision language tasks and consistently outperforms state-of-the-art methods.

Authors

Hang Li

3 papers

Yan Zeng

1 papers

Xinsong Zhang

1 papers

TL;DR

Abstract

Authors

References57 items

Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training

A Survey on Green Deep Learning

VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

An Empirical Study of Training End-to-End Vision-and-Language Transformers

KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation

SimVLM: Simple Visual Language Model Pretraining with Weak Supervision

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation

E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning

VinVL: Revisiting Visual Representations in Vision-Language Models

MDETR - Modulated Detection for End-to-End Multi-Modal Understanding

Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

Learning Transferable Visual Models From Natural Language Supervision

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

Unifying Vision-and-Language Tasks via Text Generation

UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning

Training data-efficient image transformers & distillation through attention

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Deformable DETR: Deformable Transformers for End-to-End Object Detection

Large-Scale Adversarial Training for Vision-and-Language Representation Learning

End-to-End Object Detection with Transformers

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

In Defense of Grid Features for Visual Question Answering

12-in-1: Multi-Task Vision and Language Representation Learning

Objects365: A Large-Scale, High-Quality Dataset for Object Detection

UNITER: UNiversal Image-TExt Representation Learning

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

VisualBERT: A Simple and Performant Baseline for Vision and Language

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression

The Open Images Dataset V4

A Corpus for Reasoning about Natural Language Grounded in Photographs

Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

MAttNet: Modular Attention Network for Referring Expression Comprehension

Decoupled Weight Decay Regularization

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Attention is All you Need

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Self-Critical Sequence Training for Image Captioning

Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

Modeling Context in Referring Expressions

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

Microsoft COCO Captions: Data Collection and Evaluation Server

Deep visual-semantic alignments for generating image descriptions

Microsoft COCO: Common Objects in Context

Im2Text: Describing Images Using 1 Million Captioned Photographs

Green

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

of the Association for Computational Linguistics

Field of Study

Venue Information

Name

Type

URL

Alternate Names