On the Efficacy of 3D Point Cloud Reinforcement Learning (2023-06-11T00:00:00.000000Z)

TL;DR

3D point cloud RL can significantly outperform the 2D counterpart when agent-object / object-object relationship encoding is a key factor, leading to the development of a robust algorithm for various robotic manipulation and control tasks.

Abstract

Recent studies on visual reinforcement learning (visual RL) have explored the use of 3D visual representations. However, none of these work has systematically compared the efficacy of 3D representations with 2D representations across different tasks, nor have they analyzed 3D representations from the perspective of agent-object / object-object relationship reasoning. In this work, we seek answers to the question of when and how do 3D neural networks that learn features in the 3D-native space provide a beneficial inductive bias for visual RL. We specifically focus on 3D point clouds, one of the most common forms of 3D representations. We systematically investigate design choices for 3D point cloud RL, leading to the development of a robust algorithm for various robotic manipulation and control tasks. Furthermore, through comparisons between 2D image vs 3D point cloud RL methods on both minimalist synthetic tasks and complex robotic manipulation tasks, we find that 3D point cloud RL can significantly outperform the 2D counterpart when agent-object / object-object relationship encoding is a key factor.

Authors

Z. Ling

4 papers

Xuanlin Li

6 papers

Yuan Yao

4 papers

TL;DR

Abstract

Authors

References62 items

Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation

Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Clouds

Visual Reinforcement Learning With Self-Supervised 3D Representations

Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation

A Generalist Agent

FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects

Learning Generalizable Dexterous Manipulation from Human Grasp Affordance

Learning to Detect Mobile Objects from LiDAR Scans Without Labels

R3M: A Universal Visual Representation for Robot Manipulation

Masked Visual Pre-training for Motor Control

Temporal Difference Learning for Model Predictive Control

The Unsurprising Effectiveness of Pre-Trained Vision Models for Control

SoftGroup for 3D Instance Segmentation on Point Clouds

BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning

Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation

AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions

A System for General In-Hand Object Re-Orientation

Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning

ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations

Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning

Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation

O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning

Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation

Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes

FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation

Point Transformer

Transporter Networks: Rearranging the Visual World for Robotic Manipulation

Mastering Atari with Discrete World Models

Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution

Planning to Explore via Self-Supervised World Models

Reinforcement Learning with Augmented Data

Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels

CURL: Contrastive Unsupervised Representations for Reinforcement Learning

Mastering Complex Control in MOBA Games with Deep Reinforcement Learning

Dota 2 with Large Scale Deep Reinforcement Learning

Dream to Control: Learning Behaviors by Latent Imagination

PVN3D: A Deep Point-Wise 3D Keypoints Voting Network for 6DoF Pose Estimation

RoboNet: Large-Scale Multi-Robot Learning

Improving Sample Efficiency in Model-Free Reinforcement Learning from Images

4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

KPConv: Flexible and Deformable Convolution for Point Clouds

Active Domain Randomization

DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion

Pseudo-LiDAR From Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving

Soft Actor-Critic Algorithms and Applications

Learning Latent Dynamics for Planning from Pixels

QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

Reinforcement and Imitation Learning for Diverse Visuomotor Skills

DeepMind Control Suite

Sim-to-Real Transfer of Robotic Control with Dynamics Randomization

Asymmetric Actor Critic for Image-Based Robot Learning

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Domain randomization for transferring deep neural networks from simulation to the real world

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Playing Atari with Deep Reinforcement Learning

MuJoCo: A physics engine for model-based control

The Optimal Control of Partially Observable Markov Processes over a Finite Horizon

We employ the following Point cloud post-processing procedure for training point cloud-based agents

All points with a depth larger than 6 meters are dropped from the observations 2. Each point cloud observation is uniformly downsampled to contain 128 total pints

We set the output dimension of both CNN and PointNet encoders to 128

PushChair

We reduce the dimensions of the 3 shared MLP layers of the PointNet to

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names