ProTo: Program-Guided Transformer for Program-Guided Tasks (2021-10-02T00:00:00.000000Z)

TL;DR

The Program-guided Transformer (ProTo) is proposed, which integrates both semantic and structural guidance of a program by leveraging cross-attention and masked self-att attention to pass messages between the specification and routines in the program.

Abstract

Programs, consisting of semantic and structural information, play an important role in the communication between humans and agents. Towards learning general program executors to unify perception, reasoning, and decision making, we formulate program-guided tasks which require learning to execute a given program on the observed task specification. Furthermore, we propose the Program-guided Transformer (ProTo), which integrates both semantic and structural guidance of a program by leveraging cross-attention and masked self-attention to pass messages between the specification and routines in the program. ProTo executes a program in a learned latent space and enjoys stronger representation ability than previous neural-symbolic approaches. We demonstrate that ProTo significantly outperforms the previous state-of-the-art methods on GQA visual reasoning and 2D Minecraft policy learning datasets. Additionally, ProTo demonstrates better generalization to unseen, complex, and human-written programs.

Authors

Le Song

8 papers

Binghong Chen

2 papers

Zelin Zhao

3 papers

TL;DR

Abstract

Authors

References107 items

Learning to Synthesize Programs as Interpretable and Generalizable Policies

Container: Context Aggregation Network

Dual-stream Network for Visual Recognition

ResT: An Efficient Transformer for Visual Recognition

Code Structure–Guided Transformer for Source Code Summarization

Mask Attention Networks: Rethinking and Strengthen Transformer

How to Design Sample and Computationally Efficient VQA Models

Object-Centric Diagnosis of Visual Reasoning

MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers

Learning to Execute Programs with Instruction Pointer Attention Graph Neural Networks

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Compositional Generalization via Neural-Symbolic Stack Machines

One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control

End-to-End Object Detection with Transformers

Program Guided Agent

Neural Symbolic Reader: Scalable Integration of Distributed and Symbolic Representations for Reading Comprehension

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Perceptual Visual Reasoning with Knowledge Propagation

Meta Module Network for Compositional Visual Reasoning

Program-Guided Image Manipulators

Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Learning by Abstraction: The Neural State Machine

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Natural teaching for humanoid robot via human-in-the-loop scene-motion cross-modal perception

Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning

The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

VideoBERT: A Joint Model for Video and Language Representation Learning

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

Learning to Infer and Execute 3D Shape Programs

Estimating 6D Pose From Localizing Designated Surface Keypoints

Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding

Execution-Guided Neural Program Synthesis

Learning to Describe Scenes with Programs

Explainable Neural Computation via Stack Neural Module Networks

Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies

Building Machines that Learn and Think Like People

Neural Program Synthesis from Diverse Demonstration Videos

Human-robot Interaction Oriented Human-in-the-loop Real-time Motion Imitation on a Humanoid Tri-Co Robot

Speaker-Follower Models for Vision-and-Language Navigation

Mask-Guided Contrastive Attention Model for Person Re-identification

DeepProbLog: Neural Probabilistic Logic Programming

Verifiable Reinforcement Learning via Policy Extraction

Parametrized Hierarchical Procedures for Neural Programming

Programmatically Interpretable Reinforcement Learning

Compositional Attention Networks for Machine Reasoning

Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis

Towards Specification-Directed Program Repair

Improving the Universality and Learnability of Neural Programmer-Interpreters with Combinator Abstraction

Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments

Neural-Symbolic Learning and Reasoning: A Survey and Interpretation

Graph Attention Networks

Neural Program Meta-Induction

Learning to Infer Graphics Programs from Hand-Drawn Images

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Distral: Robust multitask reinforcement learning

Program Synthesis

Neural Scene De-rendering

Towards Deep Learning Models Resistant to Adversarial Attacks

Attention is All you Need

Towards Synthesizing Complex Programs From Input-Output Examples

Inferring and Executing Programs for Visual Reasoning

RobustFill: Neural Program Learning under Noisy I/O

CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

Differentiable Functional Program Interpreters

Latent Attention For If-Then Program Synthesis

Generative Adversarial Imitation Learning

Hierarchical Question-Image Co-Attention for Visual Question Answering

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Mastering the game of Go with deep neural networks and tree search

Human-level concept learning through probabilistic program induction

Simple Baseline for Visual Question Answering

Neural GPUs Learn Algorithms

Where to Look: Focus Regions for Visual Question Answering