Learning from Physical Human Feedback: An Object-Centric One-Shot Adaptation Method (2022-03-09T00:00:00.000000Z)

TL;DR

Object Preference Adaptation is a fast, yet simple adaptation that occurs online, requires only one human intervention (one-shot), and produces new behaviors never seen during training.

Abstract

For robots to be effectively deployed in novel environments and tasks, they must be able to understand the feedback expressed by humans during intervention. This can either correct undesirable behavior or indicate additional preferences. Existing methods either require repeated episodes of interactions or assume prior known reward features, which is data-inefficient and can hardly transfer to new tasks. We relax these assumptions by describing human tasks in terms of object-centric sub-tasks and interpreting physical interventions in relation to specific objects. Our method, Object Preference Adaptation (OPA), is composed of two key stages: 1) pre-training a base policy to produce a wide variety of behaviors, and 2) online-updating according to human feedback. The key to our fast, yet simple adaptation is that general interaction dynamics between agents and objects are fixed, and only object-specific preferences are updated. Our adaptation occurs online, requires only one human intervention (one-shot), and produces new behaviors never seen during training. Trained on cheap synthetic data instead of expensive human demonstrations, our policy correctly adapts to human perturbations on realistic tasks on a physical 7DOF robot. Videos, code, and supplementary material: https://alvinosaur.github.io/AboutMe/projects/opa.

Authors

Alvin C M Shek

1 papers

Rui Chen

1 papers

Changliu Liu

1 papers

TL;DR

Abstract

Authors

References46 items

Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human–Robot Interaction

Inducing structure in reward learning by learning features

RMA: Rapid Motor Adaptation for Legged Robots

Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation

Multi-Loss Weighting with Coefficient of Variations

Learning from Interventions: Human-robot interaction as both explicit and implicit feedback

Feature Expansive Reward Learning: Rethinking Human Input

Visual Relationship Detection using Scene Graphs: A Survey

Learning to Simulate Complex Physics with Graph Networks

Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences

Learning the Correct Robot Trajectory in Real-Time from Physical Human Interactions

Graph-Structured Visual Imitation

Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations

Risk-Aware Active Inverse Reinforcement Learning

Teaching Inverse Reinforcement Learners via Features and Demonstrations

Learning under Misspecified Objective Spaces

A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress

Including Uncertainty when Learning from Human Corrections

Graph networks as learnable physics engines for inference and control

Relational inductive biases, deep learning, and graph networks

On the Convergence of Adam and Beyond

Learning Robust Rewards with Adversarial Inverse Reinforcement Learning

Learning Robot Objectives from Physical Human Interaction

Deep Reinforcement Learning from Human Preferences

Domain randomization for transferring deep neural networks from simulation to the real world

Preparing for the Unknown: Learning a Universal Policy with Online System Identification

Interaction Networks for Learning about Objects, Relations and Physics

Watch this: Scalable cost-function learning for path planning in urban environments

Learning to learn by gradient descent by gradient descent

Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization

Learning preferences for manipulation tasks from online coactive feedback

Deep Learning of Representations for Unsupervised and Transfer Learning

Feature Construction for Inverse Reinforcement Learning

A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Metrics for 3D Rotations: Comparison and Analysis

Active Learning for Reward Estimation in Inverse Reinforcement Learning

Maximum Entropy Inverse Reinforcement Learning

Apprenticeship learning via inverse reinforcement learning

Elastic bands: connecting path planning and control

Impedance Control: An Approach to Manipulation: Part II—Implementation

Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity

Learning to learn by gradient descent by gradient descent

Association for Computing Machinery

Pharmacokinetics of a novel formulation of ivermectin after administration to goats

On the momentum term in gradient descent learning algorithms

Potential Field Methods, pages 295–355

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names