Instruction-driven history-aware policies for robotic manipulations (2022-09-11T00:00:00.000000Z)

TL;DR

This work proposes a unified transformer-based approach that takes into account multiple inputs, integrates natural language instructions and multi-view scene observations and improves manipulation precision using multiple views and outperforms the state of the art.

Abstract

In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions. Yet, robotic manipulation is extremely challenging as it requires fine-grained motor control, long-term memory as well as generalization to previously unseen tasks and environments. To address these challenges, we propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations while (iii) keeping track of the full history of observations and actions. Such an approach enables learning dependencies between history and instructions and improves manipulation precision using multiple views. We evaluate our method on the challenging RLBench benchmark and on a real-world robot. Notably, our approach scales to 74 diverse RLBench tasks and outperforms the state of the art. We also address instruction-conditioned tasks and demonstrate excellent generalization to previously unseen variations.

Authors

I. Laptev

24 papers

C. Schmid

33 papers

Makarand Tapaswi

10 papers

TL;DR

Abstract

Authors

References89 items

A Generalist Agent

Coarse-to-Fine Q-attention with Learned Path Ranking

LISA: Learning Interpretable Skill Abstractions from Language

Auto-Lambda: Disentangling Dynamic Task Relationships

BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning

CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks

Simple but Effective: CLIP Embeddings for Embodied AI

Semantically Grounded Object Matching for Robust Robotic Scene Rearrangement

History Aware Multimodal Transformer for Vision-and-Language Navigation

StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects

CLIPort: What and Where Pathways for Robotic Manipulation

Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation

Implicit Behavioral Cloning

Airbert: In-domain Pretraining for Vision-and-Language Navigation

Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation

Decision Transformer: Reinforcement Learning via Sequence Modeling

Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation

Episodic Transformer for Vision-and-Language Navigation

ManipulaTHOR: A Framework for Visual Object Manipulation

MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale

Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos

Learning Transferable Visual Models From Natural Language Supervision

Composing Pick-and-Place Tasks By Grounding Language

Multi-Task Reinforcement Learning with Context-based Representations

Deep Learning in Robotics: Survey on Model Structures and Training Strategies

Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation

Learning Object Manipulation Skills via Approximate State Estimation from Real Videos

Transporter Networks: Rearranging the Visual World for Robotic Manipulation

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning

CosyPose: Consistent multi-view multi-object 6D pose estimation

Concept2Robot: Learning manipulation concepts from instructions and human demonstrations

Robot Object Retrieval with Contextual Natural Language Queries

dm_control: Software and Tasks for Continuous Control

Language Conditioned Imitation Learning Over Unstructured Data

SAPIEN: A SimulAted Part-Based Interactive ENvironment

Momentum Contrast for Unsupervised Visual Representation Learning

Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning

Solving Rubik's Cube with a Robot Hand

RLBench: The Robot Learning Benchmark & Learning Environment

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Learning to combine primitive skills: A step towards versatile robotic manipulation §

Multilingual Universal Sentence Encoder for Semantic Retrieval

Disentangled Relational Representations for Explaining and Learning from Demonstration

Which Tasks Should Be Learned Together in Multi-task Learning?

Learning Latent Plans from Play

Soft Actor-Critic Algorithms and Applications

Modulated Policy Hierarchies

Training Deep Neural Networks for Visual Servoing

Behavioral Cloning from Observation

One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning

One-Shot Visual Imitation Learning via Meta-Learning

FiLM: Visual Reasoning with a General Conditioning Layer

Grounded Language Learning in a Simulated 3D World

Attention is All you Need

Mapping Instructions and Visual Observations to Actions with Reinforcement Learning

Learning Visual Servoing with Deep Features and Fitted Q-Iteration

Deep visual foresight for planning robot motion

Learning modular neural network policies for multi-task and multi-robot transfer

Layer Normalization

Efficient Grounding of Abstract Spatial Concepts for Natural Language Interaction with Robot Manipulators

Generative Adversarial Imitation Learning

OpenAI Gym

Online Multi-Task Learning for Policy Gradient Methods

Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation

AprilTag: A robust and flexible visual fiducial system

A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Et al

Reinforcement Learning for Humanoid Robotics

LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation

Pretraining for Language-Conditioned Imitation with Transformers

Guiding Multi-Step Rearrangement Tasks with Natural Language Instructions

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding