Near-Optimal Representation Learning for Hierarchical Reinforcement Learning (2018-10-02T00:00:00.000000Z)

TL;DR

Results on a number of difficult continuous-control tasks show that the developed notion of sub-optimality of a representation, defined in terms of expected reward of the optimal hierarchical policy using this representation, yields qualitatively better representations as well as quantitatively better hierarchical policies compared to existing methods.

Abstract

We study the problem of representation learning in goal-conditioned hierarchical reinforcement learning. In such hierarchical structures, a higher-level controller solves tasks by iteratively communicating goals which a lower-level policy is trained to reach. Accordingly, the choice of representation -- the mapping of observation space to goal space -- is crucial. To study this problem, we develop a notion of sub-optimality of a representation, defined in terms of expected reward of the optimal hierarchical policy using this representation. We derive expressions which bound the sub-optimality and show how these expressions can be translated to representation learning objectives which may be optimized in practice. Results on a number of difficult continuous-control tasks show that our approach to representation learning yields qualitatively better representations as well as quantitatively better hierarchical policies, compared to existing methods (see videos at this https URL).

Authors

S. Levine

59 papers

Honglak Lee

21 papers

Ofir Nachum

6 papers

TL;DR

Abstract

Authors

References35 items

Learning deep representations by mutual information estimation and maximization

Representation Learning with Contrastive Predictive Coding

Data-Efficient Hierarchical Reinforcement Learning

MINE: Mutual Information Neural Estimation

Learning Multi-Level Hierarchies with Hindsight

Hierarchical Actor-Critic

Constrained Policy Optimization

FeUdal Networks for Hierarchical Reinforcement Learning

Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates

The Option-Critic Architecture

Near Optimal Behavior via Approximate State Abstraction

Early Visual Concept Learning with Unsupervised Deep Learning

Continuous control with deep reinforcement learning

Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images

Trust Region Policy Optimization

PAC-inspired Option Discovery in Lifelong Reinforcement Learning

Auto-Encoding Variational Bayes

MuJoCo: A physics engine for model-based control

Motor primitive discovery

An information-theoretic approach to curiosity-driven reinforcement learning

Transfer via soft homomorphisms

Performance Loss Bounds for Approximate Value Iteration with State Aggregation

Model Minimization in Hierarchical Reinforcement Learning

Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning

Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition

Reinforcement Learning with Hierarchies of Machines

Model Reduction Techniques for Computing Approximately Optimal Solutions for Markov Decision Processes

Model Minimization in Markov Decision Processes

Planning simple trajectories using neural subgoal generators

Feudal Reinforcement Learning

Aggregation and Disaggregation Techniques and Methodology in Optimization

Adaptive aggregation methods for infinite horizon dynamic programming

Approximations of Dynamic Programs, I

Towards a Unified Theory of State Abstraction for MDPs

Recent Advances in Hierarchical Reinforcement Learning

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names