You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration (2022-01-30T00:00:00.000000Z)

TL;DR

This work proposes a novel, category-level manipulation framework that leverages an object-centric, categories-level representation and model-free 6 DoF motion tracking and allows to teach different manipulation strategies by solely providing a single demonstration, without complicated manual programming.

Abstract

Promising results have been achieved recently in category-level manipulation that generalizes across object instances. Nevertheless, it often requires expensive real-world data collection and manual specification of semantic keypoints for each object category and task. Additionally, coarse keypoint predictions and ignoring intermediate action sequences hinder adoption in complex manipulation tasks beyond pick-and-place. This work proposes a novel, category-level manipulation framework that leverages an object-centric, category-level representation and model-free 6 DoF motion tracking. The canonical object representation is learned solely in simulation and then used to parse a category-level, task trajectory from a single demonstration video. The demonstration is reprojected to a target trajectory tailored to a novel object via the canonical representation. During execution, the manipulation horizon is decomposed into longrange, collision-free motion and last-inch manipulation. For the latter part, a category-level behavior cloning (CatBC) method leverages motion tracking to perform closed-loop control. CatBC follows the target trajectory, projected from the demonstration and anchored to a dynamically selected category-level coordinate frame. The frame is automatically selected along the manipulation horizon by a local attention mechanism. This framework allows to teach different manipulation strategies by solely providing a single demonstration, without complicated manual programming. Extensive experiments demonstrate its efficacy in a range of challenging industrial tasks in highprecision assembly, which involve learning complex, long-horizon policies. The process exhibits robustness against uncertainty due to dynamics as well as generalization across object instances and scene configurations. The supplementary video is available at https://www.youtube.com/watch?v=WAr8ZY3mYyw

References65 items

Learning Sensorimotor Primitives of Sequential Manipulation Tasks from Visual Demonstrations

Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation

CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation

BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models

Vision-driven Compliant Manipulation for Reliable, High-Precision Assembly Tasks

Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains

Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration

Learning Multi-Object Dense Descriptor for Autonomous Goal-Conditioned Grasping

Benchmarking Off-The-Shelf Solutions to Robotic Assembly Tasks

Robotic Pick-and-Place With Uncertain Object Instance Segmentation and Shape Completion

kPAM 2.0: Feedback Control for Category-Level Robotic Manipulation

Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos

Intermittent Visual Servoing: Efficiently Learning Policies Robust to Instrument Changes for High-precision Surgical Manipulation

S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency

Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning

se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains

Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations

SQUIRL: Robust and Efficient Learning from Video Demonstration of Long-Horizon Robotic Manipulation Tasks

PyTorch: An Imperative Style, High-Performance Deep Learning Library

RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds

Visual Geometric Skill Inference by Watching Human Demonstration

Multi-step Pick-and-Place Tasks Using Object-centric Dense Correspondences

KETO: Learning Keypoint Representations for Tool Manipulation

6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints

Scene-level Pose Estimation for Multiple Instances of Densely Packed Objects

Self-Supervised Correspondence in Visuomotor Policy Learning

Graph-Structured Visual Imitation

PoseRBPF: A Rao–Blackwellized Particle Filter for 6-D Object Pose Tracking

kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation

Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation

Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control

Learning dexterous in-hand manipulation

Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration

Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation

SE3-Pose-Nets: Structured Deep Dynamics Models for Visuomotor Control

Synthetically Trained Neural Networks for Learning Human-Readable Plans from Real-World Demonstrations

Robot learning of industrial assembly task via human demonstrations

An Algorithmic Perspective on Imitation Learning

One-Shot Visual Imitation Learning via Meta-Learning

Grasp Pose Detection in Point Clouds

Learning a visuomotor controller for real world robotic grasping using simulated depth images

Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

One-Shot Imitation Learning

Mask R-CNN

Domain randomization for transferring deep neural networks from simulation to the real world

Real-Time Perception Meets Reactive Motion Generation

Combining self-supervised learning and imitation for vision-based rope manipulation

Third-Person Imitation Learning

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection

End-to-End Training of Deep Visuomotor Policies

Probabilistic model-based imitation learning

Sampling-based algorithms for optimal motion planning

Statistical Learning by Imitation of Competing Constraints in Joint Space and Task Space

Dynamic Imitation in a Humanoid Robot through Nonparametric Probabilistic Inference

Learning Movement Primitives

A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise

Shape Modeling with Front Propagation: A Level Set Approach

Least-Squares Estimation of Transformation Parameters Between Two Point Patterns

PyTorch

Intermittent Visual Servoing: Effciently Learning Policies Robust to Instrument Changes for High- precision Surgical Manipulation

Community, Blender - a 3D modelling and rendering package, Blender Foundation

Learning from Demonstrations Through the Use of Non-rigid Registration

Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography