Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022-10-17T00:00:00.000000Z)

TL;DR

This work first generate question-guided informative image captions, and pass the captions to a PLM as context for question answering, and achieves state-of-the-art results on zero-shot VQAv2 and GQA.

Abstract

Visual question answering (VQA) is a hallmark of vision and language reasoning and a challenging task under the zero-shot setting. We propose Plug-and-Play VQA (PNP-VQA), a modular framework for zero-shot VQA. In contrast to most existing works, which require substantial adaptation of pretrained language models (PLMs) for the vision modality, PNP-VQA requires no additional training of the PLMs. Instead, we propose to use natural language and network interpretation as an intermediate representation that glues pretrained models together. We first generate question-guided informative image captions, and pass the captions to a PLM as context for question answering. Surpassing end-to-end trained baselines, PNP-VQA achieves state-of-the-art results on zero-shot VQAv2 and GQA. With 11B parameters, it outperforms the 80B-parameter Flamingo model by 8.5% on VQAv2. With 738M PLM parameters, PNP-VQA achieves an improvement of 9.1% on GQA over FewVLM with 740M PLM parameters. Code is released at https://github.com/salesforce/LAVIS/tree/main/projects/pnp-vqa

Authors

S. Hoi

19 papers

S. Savarese

31 papers

Junnan Li

7 papers

TL;DR

Abstract

Authors

References57 items

LAVIS: A Library for Language-Vision Intelligence

Least-to-Most Prompting Enables Complex Reasoning in Large Language Models

All You May Need for VQA are Image Captions

Flamingo: a Visual Language Model for Few-Shot Learning

GPT-NeoX-20B: An Open-Source Autoregressive Language Model

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

Natural Language Deduction through Search over Statement Compositions

VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning

Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models

Multitask Prompted Training Enables Zero-Shot Task Generalization

AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts

An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA

Finetuned Language Models Are Zero-Shot Learners

SimVLM: Simple Visual Language Model Pretraining with Weak Supervision

On the Opportunities and Risks of Foundation Models

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation

Multimodal Few-Shot Learning with Frozen Language Models

On Attention Redundancy: A Comprehensive Study

Learning Transferable Visual Models From Natural Language Supervision

VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Unifying Vision-and-Language Tasks via Text Generation

UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning

WeaQA: Weak Supervision via Captions for Visual Question Answering

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering

Language Models are Few-Shot Learners

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

UNITER: UNiversal Image-TExt Representation Learning

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

VisualBERT: A Simple and Performant Baseline for Vision and Language

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

RoBERTa: A Robustly Optimized BERT Pretraining Approach

OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge

The Curious Case of Neural Text Degeneration

GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

Volume 33

Language GANs Falling Short

Hierarchical Neural Story Generation

Learning with Latent Language

Controlling Linguistic Style Aspects in Neural Language Generation

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models

The modular and integrative functional architecture of the human brain

Microsoft COCO: Common Objects in Context

Modularity, comparative cognition and human uniqueness

The Modularity of mind. An essay on faculty psychology

Few-shot image classification by generating natural language rules

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Computational Linguistics (Volume 1

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names