BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022-01-28T00:00:00.000000Z)

TL;DR

BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones, and demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner.

Abstract

Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner. Code, models, and datasets are released at https://github.com/salesforce/BLIP.

Authors

S. Hoi

19 papers

Junnan Li

7 papers

Caiming Xiong

15 papers

TL;DR

Abstract

Authors

References60 items

Align and Prompt: Video-and-Language Pre-training with Entity Prompts

Scaling Up Vision-Language Pretraining for Image Captioning

LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs

VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding

SimVLM: Simple Visual Language Model Pretraining with Weak Supervision

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

Learning Transferable Visual Models From Natural Language Supervision

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Is Space-Time Attention All You Need for Video Understanding?

Unifying Vision-and-Language Tasks via Text Generation

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

VinVL: Making Visual Representations Matter in Vision-Language Models

UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning

Training data-efficient image transformers & distillation through attention

Just Ask: Learning to Answer Questions from Millions of Narrated Videos

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Support-set bottlenecks for video-text representation learning

Large-Scale Adversarial Training for Vision-and-Language Representation Learning

ActBERT: Learning Global-Local Video-Text Representations

VD-BERT: A Unified Vision and Dialog Transformer with BERT

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanations

G-DAug: Generative Data Augmentation for Commonsense Reasoning

Data Augmentation using Pre-trained Transformer Models

Hierarchical Conditional Relation Networks for Video Question Answering

Training Question Answering Models from Synthetic Data

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

End-to-End Learning of Visual Representations From Uncurated Instructional Videos

Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Self-Training With Noisy Student Improves ImageNet Classification

Do Not Have Enough Data? Deep Learning to the Rescue!

UNITER: UNiversal Image-TExt Representation Learning

Unified Vision-Language Pre-Training for Image Captioning and VQA

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

A survey on Image Data Augmentation for Deep Learning

The Curious Case of Neural Text Degeneration

Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering

Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog

A Corpus for Reasoning about Natural Language Grounded in Photographs

Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

Bilinear Attention Networks

Decoupled Weight Decay Regularization

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Visual Dialog

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

VQA: Visual Question Answering

Distilling the Knowledge in a Neural Network

Deep visual-semantic alignments for generating image descriptions

Microsoft COCO: Common Objects in Context

Im2Text: Describing Images Using 1 Million Captioned Photographs

GraDA: Graph Generative Data Augmentation for Commonsense Reasoning

83k/41k/81k images for training/validation/test

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

nocaps: novel object captioning at scale

BLIP: Bootstrapping Language-Image Pre-training for Uniﬁed Vision-Language Understanding and Generation

Field of Study

Venue Information

Name

Type

URL

Alternate Names