3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (2024-02-16T00:00:00.000000Z)

TL;DR

3D Diffuser Actor is presented, a neural policy equipped with a novel 3D denoising transformer that fuses information from the 3D visual scene, a language instruction and proprioception to predict the noise in noised 3D robot pose trajectories and its design choices dramatically outperform 2D representations, regression and classification objectives, absolute attentions, and holistic non-tokenized 3D scene embeddings.

Abstract

Diffusion policies are conditional diffusion models that learn robot action distributions conditioned on the robot and environment state. They have recently shown to outperform both deterministic and alternative action distribution learning formulations. 3D robot policies use 3D scene feature representations aggregated from a single or multiple camera views using sensed depth. They have shown to generalize better than their 2D counterparts across camera viewpoints. We unify these two lines of work and present 3D Diffuser Actor, a neural policy equipped with a novel 3D denoising transformer that fuses information from the 3D visual scene, a language instruction and proprioception to predict the noise in noised 3D robot pose trajectories. 3D Diffuser Actor sets a new state-of-the-art on RLBench with an absolute performance gain of 18.1% over the current SOTA on a multi-view setup and an absolute gain of 13.1% on a single-view setup. On the CALVIN benchmark, it improves over the current SOTA by a 9% relative increase. It also learns to control a robot manipulator in the real world from a handful of demonstrations. Through thorough comparisons with the current SOTA policies and ablations of our model, we show 3D Diffuser Actor's design choices dramatically outperform 2D representations, regression and classification objectives, absolute attentions, and holistic non-tokenized 3D scene embeddings.

Authors

Katerina Fragkiadaki

6 papers

Tsung-Wei Ke

1 papers

Nikolaos Gkanatsios

2 papers

TL;DR

Abstract

Authors

References127 items

Octo: An Open-Source Generalist Robot Policy

Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Following

Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3D

Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation

Vision-Language Foundation Models as Effective Robot Imitators

Cold Diffusion on the Replay Buffer: Learning to Plan from Known Good States

Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models

Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0

PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation

Compositional Foundation Models for Hierarchical Planning

Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Models

Diffusion-EDFs: Bi-Equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation

GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement

Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation

RVT: Robotic View Transformer for 3D Object Manipulation

DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observability

Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning

Analogy-Forming Transformers for Few-Shot 3D Parsing

Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangement

IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies

Goal-Conditioned Imitation Learning using Score-based Diffusion Policies

Diffusion policy: Visuomotor policy learning via action diffusion

ReorientDiff: Diffusion Model based Reorientation for Object Manipulation

GenAug: Retargeting behaviors to unseen situations via Generative Augmentation

AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners

Learning Universal Policies via Text-Guided Video Generation

Imitating Human Behaviour with Diffusion Models

RT-1: Robotics Transformer for Real-World Control at Scale

CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learning

InstructPix2Pix: Learning to Follow Image Editing Instructions

From Play to Policy: Conditional Behavior Generation from Uncurated Robot Data

LAION-5B: An open large-scale dataset for training next generation image-text models

DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics

Grounding Language with Visual Affordances over Unstructured Data

Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling

Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation

Instruction-driven history-aware policies for robotic manipulations

SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion

Goal-Aware Generative Adversarial Imitation Learning from Imperfect Demonstration for Robotic Cloth Manipulation

Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning

Conditional Energy-Based Models for Implicit Policies: The Gap between Theory and Practice

Behavior Transformers: Cloning k modes with one stone

PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies

Planning with Diffusion for Flexible Behavior Synthesis

A Generalist Agent

What Matters in Language Conditioned Robotic Imitation Learning Over Unstructured Data

BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning

CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks

Lepard: Learning partial point cloud matching in rigid and deformable scenes

CLIPort: What and Where Pathways for Robotic Manipulation

Implicit Behavioral Cloning

Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation

Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation

RoFormer: Enhanced Transformer with Rotary Position Embedding

Perceiver: General Perception with Iterative Attention

Learning Transferable Visual Models From Natural Language Supervision

Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks

Transporter Networks: Rearranging the Visual World for Robotic Manipulation

Denoising Diffusion Implicit Models

Denoising Diffusion Probabilistic Models

Language Conditioned Imitation Learning Over Unstructured Data

Grounding Language in Play

IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data

RLBench: The Robot Learning Benchmark & Learning Environment

Goal-conditioned Imitation Learning

On the Continuity of Rotation Representations in Neural Networks

Self-Attention with Relative Position Representations

FiLM: Visual Reasoning with a General Conditioning Layer

Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets

Feature Pyramid Networks for Object Detection

Generative Adversarial Imitation Learning

End to End Learning for Self-Driving Cars

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Study