Large-Scale Adversarial Training for Vision-and-Language Representation Learning

Published in

Neural Information Processing Systems(2020)

External Links:

Generate Graph

TL;DR

To enable large-scale training, VILLA adopts the "free" adversarial training strategy, and combines it with KL-divergence-based regularization to promote higher invariance in the embedding space.

Abstract

We present VILLA, the first known effort on large-scale adversarial training for vision-and-language (V+L) representation learning. VILLA consists of two training stages: (i) task-agnostic adversarial pre-training; followed by (ii) task-specific adversarial finetuning. Instead of adding adversarial perturbations on image pixels and textual tokens, we propose to perform adversarial training in the embedding space of each modality. To enable large-scale training, we adopt the "free" adversarial training strategy, and combine it with KL-divergence-based regularization to promote higher invariance in the embedding space. We apply VILLA to current best-performing V+L models, and achieve new state of the art on a wide range of tasks, including Visual Question Answering, Visual Commonsense Reasoning, Image-Text Retrieval, Referring Expression Comprehension, Visual Entailment, and NLVR2.

Authors

Zhe Gan

19 papers

Linjie Li

14 papers

Jingjing Liu

13 papers

References92 items

Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models

Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web

VD-BERT: A Unified Vision and Dialog Transformer with BERT

Adversarial Training for Large Neural Language Models

Large-Scale Adversarial Training for Vision-and-Language Representation Learning

Published in

Neural Information Processing Systems(2020)

External Links:

Generate Graph

TL;DR

To enable large-scale training, VILLA adopts the "free" adversarial training strategy, and combines it with KL-divergence-based regularization to promote higher invariance in the embedding space.

Abstract

Authors

Zhe Gan

19 papers

Linjie Li

14 papers

Jingjing Liu

13 papers

References92 items

Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models

Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web

VD-BERT: A Unified Vision and Dialog Transformer with BERT

Adversarial Training for Large Neural Language Models

Yen-Chun Chen

2 papers

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers

Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning

XGPT: Cross-modal Generative Pre-Training for Image Captioning

Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training

Boosting Adversarial Training with Hypersphere Embedding

UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Fast is better than free: Revisiting adversarial training

12-in-1: Multi-Task Vision and Language Representation Learning

Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline

Adversarial Examples Improve Image Recognition

SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization

Meta Module Network for Compositional Visual Reasoning

FreeLB: Enhanced Adversarial Training for Natural Language Understanding

UNITER: UNiversal Image-TExt Representation Learning

UNITER: Learning UNiversal Image-TExt Representations

FreeLB: Enhanced Adversarial Training for Language Understanding

Learning Video Representations using Contrastive Bidirectional Transformer

Unified Vision-Language Pre-Training for Image Captioning and VQA

Metric Learning for Adversarial Robustness

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

VisualBERT: A Simple and Performant Baseline for Vision and Language

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Fusion of Detected Objects in Text for Visual Question Answering

Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training

Contrastive Bidirectional Transformer for Temporal Representation Learning

Deep Modular Co-Attention Networks for Visual Question Answering

Unlabeled Data Improves Adversarial Robustness

Are Labels Required for Improving Adversarial Robustness?

Improving the Robustness of Deep Neural Networks via Adversarial Training with Triplet Loss

Improving Neural Language Modeling via Adversarial Training

Exact Adversarial Attack to Image Captioning via Structured Output Learning With Latent Variables

You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle

Unsupervised Data Augmentation for Consistency Training

Adversarial Training for Free!

VideoBERT: A Joint Model for Video and Language Representation Learning

Relation-Aware Graph Attention Network for Visual Question Answering

MUREL: Multimodal Relational Reasoning for Visual Question Answering

GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

Cycle-Consistency for Robust Visual Question Answering

Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog

Using Pre-Training Can Improve Model Robustness and Uncertainty

Theoretically Principled Trade-off between Robustness and Accuracy

Visual Entailment: A Novel Task for Fine-Grained Image Understanding

Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering

Feature Denoising for Improving Adversarial Robustness

From Recognition to Cognition: Visual Commonsense Reasoning

A Corpus for Reasoning about Natural Language Grounded in Photographs

Overcoming Language Priors in Visual Question Answering with Adversarial Regularization

Semantically Equivalent Adversarial Rules for Debugging NLP models

Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

Learning Conditioned Graph Structures for Interpretable Visual Question Answering

Bilinear Attention Networks

Stacked Cross Attention for Image-Text Matching

Compositional Attention Networks for Machine Reasoning

Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples

Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning

Show-and-Fool: Crafting Adversarial Examples for Neural Image Captioning

Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Towards Deep Learning Models Resistant to Adversarial Attacks

Attention is All you Need

A simple neural network module for relational reasoning

Ensemble Adversarial Training: Attacks and Defenses

Inferring and Executing Programs for Visual Reasoning

Learning to Reason: End-to-End Module Networks for Visual Question Answering

Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Modeling Context in Referring Expressions

Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

Hierarchical Question-Image Co-Attention for Visual Question Answering

Adversarial Training Methods for Semi-Supervised Text Classification

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Neural Module Networks

Stacked Attention Networks for Image Question Answering

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

VQA: Visual Question Answering

Explaining and Harnessing Adversarial Examples

Microsoft COCO: Common Objects in Context

Intriguing properties of neural networks

Im2Text: Describing Images Using 1 Million Captioned Photographs

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Hadamard Product for Low-rank Bilinear Pooling

Implementation Details Our models are implemented based on PyTorch. 2 To speed up training, we use Nvidia Apex 3 for mixed precision training. All pre-training experiments are run on Nvidia V100 GPUs

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

Neural Information Processing Systems

Type

conference

URL

http://neurips.cc/

Alternate Names

Neural Inf Process Syst
NeurIPS
NIPS