Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Published in

European Conference on Computer Vision(2020)

External Links:

Generate Graph

TL;DR

This paper proposes a new learning method Oscar (Object-Semantics Aligned Pre-training), which uses object tags detected in images as anchor points to significantly ease the learning of alignments.

Abstract

Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatenate image region features and text features as input to the model to be pre-trained and use self-attention to learn image-text semantic alignments in a brute force manner, in this paper, we propose a new learning method Oscar (Object-Semantics Aligned Pre-training), which uses object tags detected in images as anchor points to significantly ease the learning of alignments. Our method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. We pre-train an Oscar model on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks.

Authors

Chunyuan Li

13 papers

Jianfeng Gao

39 papers

Houdong Hu

5 papers

References52 items

Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training

ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data

12-in-1: Multi-Task Vision and Language Representation Learning

Meta Module Network for Compositional Visual Reasoning

UNITER: UNiversal Image-TExt Representation Learning

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Published in

European Conference on Computer Vision(2020)

External Links:

Generate Graph

TL;DR

This paper proposes a new learning method Oscar (Object-Semantics Aligned Pre-training), which uses object tags detected in images as anchor points to significantly ease the learning of alignments.

Abstract

Authors

Chunyuan Li

13 papers

Jianfeng Gao

39 papers

Houdong Hu

5 papers

References52 items

Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training

ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data

12-in-1: Multi-Task Vision and Language Representation Learning

Meta Module Network for Compositional Visual Reasoning

UNITER: UNiversal Image-TExt Representation Learning

Lijuan Wang

16 papers

Pengchuan Zhang

8 papers

Yejin Choi

44 papers

UNITER: Learning UNiversal Image-TExt Representations

Unified Vision-Language Pre-Training for Image Captioning and VQA

CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Attention on Attention for Image Captioning

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

VisualBERT: A Simple and Performant Baseline for Vision and Language

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Knowledge Aware Semantic Concept Expansion for Image-Text Matching

Position Focused Attention Network for Image-Text Matching

Learning by Abstraction: The Neural State Machine

VideoBERT: A Joint Model for Video and Language Representation Learning

GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

Multi-Task Deep Neural Networks for Natural Language Understanding

The Open Images Dataset V4

A Corpus for Reasoning about Natural Language Grounded in Photographs

Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

Stacked Cross Attention for Image-Text Matching

Dual-path Convolutional Image-Text Embeddings with Instance Loss

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

VSE++: Improved Visual-Semantic Embeddings

VSE++: Improving Visual-Semantic Embeddings with Hard Negatives

Attention is All you Need

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Guided Open Vocabulary Image Captioning with Constrained Beam Search

Self-Critical Sequence Training for Image Captioning

Joint Image-Text Representation by Gaussian Visual-Semantic Embedding

Image Captioning with Semantic Attention

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

What Value Do Explicit High Level Concepts Have in Vision to Language Problems?

Deep visual-semantic alignments for generating image descriptions

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Microsoft COCO: Common Objects in Context

From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

Zero-Shot Learning by Convex Combination of Semantic Embeddings

DeViSE: A Deep Visual-Semantic Embedding Model

Zero-Shot Learning Through Cross-Modal Transfer

Im2Text: Describing Images Using 1 Million Captioned Photographs

Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora

Aligning Sentences in Parallel Corpora

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

nocaps: novel object captioning at scale

Visualizing Data using t-SNE

NLVR2 For the Oscar B model, we fine-tune for 20 epochs with learning rate {2e −5 , 3e −5 , 5e −5 } and a batch size of 72. For the Oscar L model

Oscar : Object - Semantics Aligned Pre - training for Vision - Language Tasks

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

European Conference on Computer Vision

Type

conference

URL

https://link.springer.com/conference/eccv

Alternate Names

ECCV
Eur Conf Comput Vis