Generative Image as Action Models

Published in

Conference on Robot Learning(2024)

External Links:

Generate Graph

Abstract

Image-generation diffusion models have been fine-tuned to unlock new capabilities such as image-editing and novel view synthesis. Can we similarly unlock image-generation models for visuomotor control? We present GENIMA, a behavior-cloning agent that fine-tunes Stable Diffusion to 'draw joint-actions' as targets on RGB images. These images are fed into a controller that maps the visual targets into a sequence of joint-positions. We study GENIMA on 25 RLBench and 9 real-world manipulation tasks. We find that, by lifting actions into image-space, internet pre-trained diffusion models can generate policies that outperform state-of-the-art visuomotor approaches, especially in robustness to scene perturbations and generalizing to novel objects. Our method is also competitive with 3D agents, despite lacking priors such as depth, keypoints, or motion-planners.

Authors

Mohit Shridhar

5 papers

Yat Long Lo

1 papers

Stephen James

1 papers

References104 items

Render and Diffuse: Aligning Image and Action Spaces for Diffusion-based Behaviour Cloning

Octo: An Open-Source Generalist Robot Policy

Enabling Stateful Behaviors for Diffusion-based Policy Learning

ALDM-Grasping: Diffusion-aided Zero-Shot Sim-to-Real Transfer for Robot Grasping

DNAct: Diffusion Guided Multi-Task 3D Policy Learning

Generative Image as Action Models

Published in

Conference on Robot Learning(2024)

External Links:

Generate Graph

Abstract

Authors

Mohit Shridhar

5 papers

Yat Long Lo

1 papers

Stephen James

1 papers

References104 items

Render and Diffuse: Aligning Image and Action Spaces for Diffusion-based Behaviour Cloning

Octo: An Open-Source Generalist Robot Policy

Enabling Stateful Behaviors for Diffusion-based Policy Learning

ALDM-Grasping: Diffusion-aided Zero-Shot Sim-to-Real Transfer for Robot Grasping

DNAct: Diffusion Guided Multi-Task 3D Policy Learning

Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation

3D Diffusion Policy

Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches

Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learning

Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training

Pedipulate: Enabling Manipulation Skills using a Quadruped Robot’s Leg

3D Diffuser Actor: Policy Diffusion with 3D Scene Representations

THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation

PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs

Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data

Simple Hierarchical Planning with Diffusion

Any-point Trajectory Modeling for Policy Learning

StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation

SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution

Photorealistic Video Generation with Diffusion Models

PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play

Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertainty

Adversarial Diffusion Distillation

Diffusion Model Alignment Using Direct Preference Optimization

Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model

RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches

Constrained-Context Conditional Diffusion Models for Imitation Learning

Cold Diffusion on the Replay Buffer: Learning to Plan from Known Good States

Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models

Video Language Planning

Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models

Compositional Foundation Models for Hierarchical Planning

RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking

MVDream: Multi-view Diffusion for 3D Generation

GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields

RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation

IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition

XSkill: Cross Embodiment Skill Discovery

SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis

Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learning

RVT: Robotic View Transformer for 3D Object Manipulation

Emergent Correspondence from Image Diffusion

Extracting Reward Functions from Diffusion Models

Unsupervised Semantic Correspondence Using Stable Diffusion

Training Diffusion Models with Reinforcement Learning

Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware

Goal-Conditioned Imitation Learning using Score-based Diffusion Policies

Affordance Diffusion: Synthesizing Hand-Object Interactions

Stable Bias: Evaluating Societal Representations in Diffusion Models

Zero-1-to-3: Zero-shot One Image to 3D Object

Diffusion policy: Visuomotor policy learning via action diffusion

Scaling Robot Learning with Semantically Imagined Experience

T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models

GenAug: Retargeting behaviors to unseen situations via Generative Augmentation

Adding Conditional Control to Text-to-Image Diffusion Models

AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners

Learning Universal Policies via Text-Guided Video Generation

Is Conditional Generative Modeling all you need for Decision-Making?

InstructPix2Pix: Learning to Follow Image Editing Instructions

Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models

StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects

LAION-5B: An open large-scale dataset for training next generation image-text models

Real-World Robot Learning with Masked Visual Pre-training

DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics

DreamFusion: Text-to-3D using 2D Diffusion

Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation

Instruction-driven history-aware policies for robotic manipulations

SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion

Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning

Elucidating the Design Space of Diffusion-Based Generative Models

Planning with Diffusion for Flexible Behavior Synthesis

High-Resolution Image Synthesis with Latent Diffusion Models

Multimodal datasets: misogyny, pornography, and malignant stereotypes

CLIPort: What and Where Pathways for Robotic Manipulation

Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning

Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation

Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation

Learning Transferable Visual Models From Natural Language Supervision

Zero-Shot Text-to-Image Generation

Transporter Networks: Rearranging the Visual World for Robotic Manipulation

Denoising Diffusion Probabilistic Models

End-to-End Object Detection with Transformers

Camera-to-Robot Pose Estimation from a Single Image

RLBench: The Robot Learning Benchmark & Learning Environment

PyRep: Bringing V-REP to Deep Robot Learning

FiLM: Visual Reasoning with a General Conditioning Layer

Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task

Attention is All you Need

3D Simulation for Robot Arm Control with Deep Q-Learning

Deep Residual Learning for Image Recognition

U-Net: Convolutional Networks for Biomedical Image Segmentation

End-to-End Training of Deep Visuomotor Policies

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Track2Act: Predicting Point Tracks from Internet Videos enables Diverse Zero-shot Robot Manipulation

ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation

100

EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior

101

Hierarchical Diffusion for Offline Decision Making

102

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

103

V-REP: A versatile and scalable robot simulation framework

104

Diffusers: State-of-the-art diffusion models

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

Conference on Robot Learning

Type

conference

URL

N/A

Alternate Names

CoRL
Conf Robot Learn