Grounded Language-Image Pre-training (2021-12-07T00:00:00.000000Z)

TL;DR

A grounded language-image pretraining model for learning object-level, language-aware, and semantic-rich visual representations that unifies object detection and phrase grounding for pre-training and can leverage massive image-text pairs by generating grounding boxes in a self-training fashion.

Abstract

This paper presents a grounded language-image pretraining (GLIP) model for learning object-level, language-aware, and semantic-rich visual representations. GLIP unifies object detection and phrase grounding for pre-training. The unification brings two benefits: 1) it allows GLIP to learn from both detection and grounding data to improve both tasks and bootstrap a good grounding model; 2) GLIP can leverage massive image-text pairs by generating grounding boxes in a self-training fashion, making the learned representations semantic-rich. In our experiments, we pre-train GLIP on 27M grounding data, including 3M human-annotated and 24M web-crawled image-text pairs. The learned representations demonstrate strong zero-shot and few-shot transferability to various object-level recognition tasks. 1) When directly evaluated on COCO and LVIS (without seeing any images in COCO during pre-training), GLIP achieves 49.8 AP and 26.9 AP, respectively, surpassing many supervised baselines.11Supervised baselines on COCO object detection: Faster-RCNN w/ ResNet50 (40.2) or ResNet101 (42.0), and DyHead w/ Swin-Tiny (49.7). 2) After fine-tuned on COCO, GLIP achieves 60.8 AP on val and 61.5 AP on test-dev, surpassing prior SoTA. 3) When transferred to 13 downstream object detection tasks, a 1-shot GLIP rivals with a fully-supervised Dynamic Head. Code will be released at https://github.com/microsoft/GLIP.

Authors

Jianfeng Gao

21 papers

Chunyuan Li

13 papers

Lu Yuan

7 papers

TL;DR

Abstract

Authors

References79 items

Omni-DETR: Omni-Supervised Object Detection with Transformers

Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

Learning to Prompt for Vision-Language Models

Learning Open-World Object Proposals without Learning to Classify

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

Image Scene Graph Generation (SGG) Benchmark

Focal Self-attention for Local-Global Interactions in Vision Transformers

End-to-End Semi-Supervised Object Detection with Soft Teacher

Dynamic Head: Unifying Object Detection Heads with Attentions

Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions

VinVL: Revisiting Visual Representations in Vision-Language Models

MDETR - Modulated Detection for End-to-End Multi-Modal Understanding

The Power of Scale for Parameter-Efficient Prompt Tuning

Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding

Towards Open World Object Detection

Learning Transferable Visual Models From Natural Language Supervision

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Open-Vocabulary Object Detection Using Captions

Deformable DETR: Deformable Transformers for End-to-End Object Detection

What leads to generalization of object proposals?

Zero-Shot Object Detection: Joint Recognition and Localization of Novel Concepts

Rethinking Pre-training and Self-training

Language Models are Few-Shot Learners

End-to-End Object Detection with Transformers

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Improved Visual-Semantic Alignment for Zero-Shot Object Detection

Frustratingly Simple Few-Shot Object Detection

The Overlooked Elephant of Object Detection: Open Set

Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs

Don’t Even Look Once: Synthesizing Features for Zero-Shot Detection

Objects365: A Large-Scale, High-Quality Dataset for Object Detection

Meta-Learning to Detect Rare Objects

Meta R-CNN: Towards General Solver for Instance-Level Low-Shot Learning

UNITER: Learning UNiversal Image-TExt Representations

Unified Vision-Language Pre-Training for Image Captioning and VQA

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

VisualBERT: A Simple and Performant Baseline for Vision and Language

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

LVIS: A Dataset for Large Vocabulary Instance Segmentation

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

FCOS: Fully Convolutional One-Stage Object Detection

GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

Deep High-Resolution Representation Learning for Human Pose Estimation

Hybrid Task Cascade for Instance Segmentation

Few-Shot Object Detection via Feature Reweighting

Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

Bilinear Attention Networks

Simple Baselines for Human Pose Estimation and Tracking

Zero-Shot Object Detection

Zero Shot Detection

Focal Loss for Dense Object Detection

Attention is All you Need

Mask R-CNN

Scene Graph Generation by Iterative Message Passing

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

R-FCN: Object Detection via Region-based Fully Convolutional Networks

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Deep Residual Learning for Image Recognition

You Only Look Once: Unified, Real-Time Object Detection

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

Fully convolutional networks for semantic segmentation

GloVe: Global Vectors for Word Representation

Microsoft COCO: Common Objects in Context

ImageNet classification with deep convolutional neural networks

Im2Text: Describing Images Using 1 Million Captioned Photographs

Steven Bird, Ewan Klein and Edward Loper: Natural Language Processing with Python, Analyzing Text with the Natural Language Toolkit

Zero-Shot Detection via Vision and Language Knowledge Distillation

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Conference on Machine Learning

The open images dataset v4: Uniﬁed image classiﬁcation, object detection, and visual relationship detection at scale