Maximum Entropy-Regularized Multi-Goal Reinforcement Learning (2019-05-21T00:00:00.000000Z)

TL;DR

A novel multi-goal RL objective based on weighted entropy is proposed, which encourages the agent to maximize the expected return, as well as to achieve more diverse goals and a maximum entropy-based prioritization framework is developed to optimize the proposed objective.

Abstract

In Multi-Goal Reinforcement Learning, an agent learns to achieve multiple goals with a goal-conditioned policy. During learning, the agent first collects the trajectories into a replay buffer, and later these trajectories are selected randomly for replay. However, the achieved goals in the replay buffer are often biased towards the behavior policies. From a Bayesian perspective, when there is no prior knowledge about the target goal distribution, the agent should learn uniformly from diverse achieved goals. Therefore, we first propose a novel multi-goal RL objective based on weighted entropy. This objective encourages the agent to maximize the expected return, as well as to achieve more diverse goals. Secondly, we developed a maximum entropy-based prioritization framework to optimize the proposed objective. For evaluation of this framework, we combine it with Deep Deterministic Policy Gradient, both with or without Hindsight Experience Replay. On a set of multi-goal robotic tasks of OpenAI Gym, we compare our method with other baselines and show promising improvements in both performance and sample-efficiency.

Authors

Volker Tresp

5 papers

Rui Zhao

2 papers

Xudong Sun

1 papers

TL;DR

Abstract

Authors

References57 items

Energy-Based Hindsight Experience Prioritization

Efficient Dialog Policy Learning via Positive Memory Retention

Curiosity-Driven Experience Prioritization via Density Estimation

Unsupervised Control Through Non-Parametric Discriminative Rewards

Visual Reinforcement Learning with Imagined Goals

Improving Goal-Oriented Visual Dialog Agents via Advanced Recurrent Nets with Tempered Policy Gradient

Learning Goal-Oriented Visual Dialog via Tempered Policy Gradient

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review

Latent Space Policies for Hierarchical Reinforcement Learning

Composable Deep Reinforcement Learning for Robotic Manipulation

Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research

Diversity is All You Need: Learning Skills without a Reward Function

Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Hindsight policy gradients

Weighted entropy: basic inequalities

Hindsight Experience Replay

A unified view of entropy-regularized Markov decision processes

Automatic Goal Generation for Reinforcement Learning Agents

Equivalence Between Policy Gradients and Soft Q-Learning

Two-stream RNN/CNN for action recognition in 3D videos

Reinforcement Learning with Deep Energy-Based Policies

Regularizing Neural Networks by Penalizing Confident Output Distributions

An Alternative Softmax Operator for Reinforcement Learning

Training Agent for First-Person Shooter Game with Actor-Critic Curriculum Learning

Improving Policy Gradient by Exploring Under-appreciated Rewards

Path integral guided policy search

Learning to push by grasping: Using multiple tasks for effective learning

Learning End-to-End Goal-Oriented Dialog

Asynchronous Methods for Deep Reinforcement Learning

Mastering the game of Go with deep neural networks and tree search

Prioritized Experience Replay

Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

Continuous control with deep reinforcement learning

Universal Value Function Approximators

End-to-End Training of Deep Visuomotor Policies

Human-level control through deep reinforcement learning

Trust Region Policy Optimization

Universal Option Models

Horde: a scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction

A Survey on Transfer Learning

Reinforcement learning of motor skills with policy gradients

Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning

Multitask Learning

Hierarchical Learning in Stochastic Domains: Preliminary Results

Self-improving reactive agents based on reinforcement learning, planning and teaching

Inequalities: Theory of Majorization and Its Applications

Deep learning , volume 1

Machine learning - a probabilistic perspective

Pearson correlation coefficient. In Noise reduction in speech processing

Pearson correlation coefﬁcient

INEQUALITIES OF KARAMATA , SCHUR AND MUIRHEAD , AND SOME APPLICATIONS

Hierarchical Reinforcement Learning Based on Subgoal Discovery and Subpolicy Specialization

Reinforcement Learning: An Introduction

Function Optimization using Connectionist Reinforcement Learning Algorithms

This Paper Is Included in the Proceedings of the 12th Usenix Symposium on Operating Systems Design and Implementation (osdi '16). Tensorflow: a System for Large-scale Machine Learning Tensorflow: a System for Large-scale Machine Learning

Shannon entropy of p(x)(1 − p(x)) is

Field of Study

Venue Information

Name

Type

URL

Alternate Names