Universal Planning Networks

Published in

International Conference on Machine Learning(2018)

External Links:

Generate Graph

TL;DR

This work finds that the representations learned are not only effective for goal-directed visual imitation via gradient-based trajectory optimization, but can also provide a metric for specifying goals using images.

Abstract

A key challenge in complex visuomotor control is learning abstract representations that are effective for specifying goals, planning, and generalization. To this end, we introduce universal planning networks (UPN). UPNs embed differentiable planning within a goal-directed policy. This planning computation unrolls a forward model in a latent space and infers an optimal action plan through gradient descent trajectory optimization. The plan-by-gradient-descent process and its underlying representations are learned end-to-end to directly optimize a supervised imitation learning objective. We find that the representations learned are not only effective for goal-directed visual imitation via gradient-based trajectory optimization, but can also provide a metric for specifying goals using images. The learned representations can be leveraged to specify distance-based rewards to reach new target states for model-free reinforcement learning, resulting in substantially more effective learning when solving new tasks described via image-based goals. We were able to achieve successful transfer of visuomotor planning strategies across robots with significantly different morphologies and actuation capabilities.

Authors

Chelsea Finn

26 papers

P. Abbeel

35 papers

S. Levine

59 papers

References69 items

MPC-Inspired Neural Network Policies for Sequential Decision Making

Zero-Shot Visual Imitation

Integrating State Representation Learning Into Deep Reinforcement Learning

PixelSNAIL: An Improved Autoregressive Generative Model

Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm

Universal Planning Networks

Published in

International Conference on Machine Learning(2018)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Chelsea Finn

26 papers

P. Abbeel

35 papers

S. Levine

59 papers

References69 items

MPC-Inspired Neural Network Policies for Sequential Decision Making

Zero-Shot Visual Imitation

Integrating State Representation Learning Into Deep Reinforcement Learning

PixelSNAIL: An Improved Autoregressive Generative Model

Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm

A. Srinivas

6 papers

TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning

TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning

Meta Learning Shared Hierarchies

Swish: a Self-Gated Activation Function

Emergent Complexity via Multi-Agent Competition

One-Shot Visual Imitation Learning via Meta-Learning

DARLA: Improving Zero-Shot Transfer in Reinforcement Learning

Proximal Policy Optimization Algorithms

Imagination-Augmented Agents for Deep Reinforcement Learning

End-to-End Differentiable Adversarial Imitation Learning

Value Prediction Network

Hindsight Experience Replay

PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations

Model-Based Planning in Discrete Action Spaces

Curiosity-Driven Exploration by Self-Supervised Prediction

Time-Contrastive Networks: Self-Supervised Learning from Video

Time-Contrastive Networks: Self-Supervised Learning from Multi-view Observation

Inferring The Latent Structure of Human Decision-Making from Raw Visual Inputs

InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations

Domain randomization for transferring deep neural networks from simulation to the real world

Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Combining self-supervised learning and imitation for vision-based rope manipulation

OptNet: Differentiable Optimization as a Layer in Neural Networks

Task-based End-to-end Model Learning in Stochastic Optimization

The Predictron: End-To-End Learning and Planning

Unsupervised Perceptual Rewards for Imitation Learning

Incorporating Human Domain Knowledge into Large Scale Cost Function Learning

(CAD)$^2$RL: Real Single-Image Flight without a Single Real Image

Layer Normalization

Learning to Poke by Poking: Experiential Learning of Intuitive Physics

Generative Adversarial Imitation Learning

The Curious Robot: Learning Visual Representations via Physical Interactions

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

Value Iteration Networks

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Deep spatial autoencoders for visuomotor learning

Learning state representations with robotic priors

Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images

End-to-End Training of Deep Visuomotor Policies

Human-level control through deep reinforcement learning

Auto-Encoding Variational Bayes

Synthesis and stabilization of complex behaviors through online trajectory optimization

MuJoCo: A physics engine for model-based control

Autonomous reinforcement learning on raw visual input data in a real world application

Apprenticeship learning via inverse reinforcement learning

Image-based simultaneous control of robot and target object motions by direct-image-interpretation method

Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping

An on-line algorithm for dynamic reinforcement learning and planning in reactive environments

Gradient Theory of Optimal Flight Paths

Searching for Activation Functions

Learning to search with MCTSnets, 2018

DARLA: Improving Zero-Shot Transfer in Reinforcement Learning

Learning from the hindsight planepisodic mpc improvement

Optimizing Expectations: From Deep Reinforcement Learning to Stochastic Computation Graphs

Deep learning , volume 1

An Introduction to Reinforcement Learning

Pharmacokinetics of a novel formulation of ivermectin after administration to goats

h = layers.convolution2d(h, num_outputs=64,kernel_size=3,stride=1,padding=’VALID’, activation_fn=None)

next_state = layers.fully_connected( tf.concat([curr_state, action]), out_dim=128, nonlinearity=None)

next_state = layers.layer_norm(next_state, center=True, scale=True

Field of Study

Computer ScienceMathematics

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

International Conference on Machine Learning

Type

conference

URL

https://icml.cc/

Alternate Names

ICML
Int Conf Mach Learn