Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Clouds (2022-10-14T00:00:00.000000Z)

Abstract

We study how choices of input point cloud coordinate frames impact learning of manipulation skills from 3D point clouds. There exist a variety of coordinate frame choices to normalize captured robot-object-interaction point clouds. We find that different frames have a profound effect on agent learning performance, and the trend is similar across 3D backbone networks. In particular, the end-effector frame and the target-part frame achieve higher training efficiency than the commonly used world frame and robot-base frame in many tasks, intuitively because they provide helpful alignments among point clouds across time steps and thus can simplify visual module learning. Moreover, the well-performing frames vary across tasks, and some tasks may benefit from multiple frame candidates. We thus propose FrameMiners to adaptively select candidate frames and fuse their merits in a task-agnostic manner. Experimentally, FrameMiners achieves on-par or significantly higher performance than the best single-frame version on five fully physical manipulation tasks adapted from ManiSkill and OCRTOC. Without changing existing camera placements or adding extra cameras, point cloud frame mining can serve as a free lunch to improve 3D manipulation learning.

Authors

Yangyan Li

5 papers

Minghua Liu

5 papers

Z. Ling

4 papers

Abstract

Authors

References62 items

FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects

Learning Generalizable Dexterous Manipulation from Human Grasp Affordance

R3M: A Universal Visual Representation for Robot Manipulation

Vision-Based Manipulators Need to Also See from Their Hands

Ditto: Building Digital Twins of Articulated Objects from Interaction

You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration

Benchmarking Robustness of 3D Point Cloud Recognition Against Common Corruptions

Look Closer: Bridging Egocentric and Third-Person Views With Transformers for Robotic Manipulation

Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation

The Surprising Effectiveness of Representation Learning for Visual Imitation

SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning

A System for General In-Hand Object Re-Orientation

Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning

ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations

RRL: Resnet as representation for Reinforcement Learning

Unsupervised Learning of Visual 3D Keypoints for Control

Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation

Learning Visible Connectivity Dynamics for Cloth Smoothing

Vector Neurons: A General Framework for SO(3)-Equivariant Networks

OCRTOC: A Cloud-Based Competition and Benchmark for Robotic Grasping and Manipulation

Robustness Certification for Point Cloud Models

Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes

AMVNet: Assertion-based Multi-View Fusion Network for LiDAR Semantic Segmentation

Multi Projection Fusion for Real-time Semantic Segmentation of 3D LiDAR Point Clouds

S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency

TORNADO-Net: mulTiview tOtal vaRiatioN semAntic segmentation with Diamond inceptiOn module

Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution

SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks

Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels

A Rotation-Invariant Framework for Deep Point Cloud Analysis

Active Perception and Representation for Robotic Manipulation

PVN3D: A Deep Point-Wise 3D Keypoints Voting Network for 6DoF Pose Estimation

End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds

Rotation Invariant Convolutions for 3D Point Clouds Deep Learning

Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds

ClusterNet: Deep Hierarchical Cluster Network With Rigorously Rotation-Invariant Representation for Point Cloud Analysis

Deep Hough Voting for 3D Object Detection in Point Clouds

DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion

PointPillars: Fast Encoders for Object Detection From Point Clouds

GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud

Reinforcement Learning of Active Vision for Manipulating Objects under Occlusions

Point convolutional neural networks by extension operators

Reinforcement and Imitation Learning for Diverse Visuomotor Skills

Tensor Field Networks: Rotation- and Translation-Equivariant Neural Networks for 3D Point Clouds

Dynamic Graph CNN for Learning on Point Clouds

PointCNN: Convolution On X-Transformed Points

Frustum PointNets for 3D Object Detection from RGB-D Data

Proximal Policy Optimization Algorithms

Attention is All you Need

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Submanifold Sparse Convolutional Networks

Intel RealSense Stereoscopic Depth Cameras

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

A Point Set Generation Network for 3D Object Reconstruction from a Single Image

ShapeNet: An Information-Rich 3D Model Repository

The YCB object and Model set: Towards common benchmarks for manipulation research

Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols

Mixture of experts: a literature survey

End-to-End Learning to Grasp from Object Point Clouds

Polosukhin. Attention is all you need. Advances in neural information processing systems

More Details of Manipulation Tasks Task Descriptions

single-arm mobile agent needs to approach a cabinet, use the handle to fully open the designated cabinet drawer, and then keep the drawer static for a while

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names