MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics

Published in

International Joint Conference on Artificial In...(2019)

External Links:

Generate Graph DownloadPDF

TL;DR

This work explores a new challenge in transfer RL, where only a set of source policies collected under diverse unknown dynamics is available for learning a target task efficiently, and proposes MULTI-source POLicy AggRegation (MULTIPOLAR), which learns to aggregate the actions provided by the source policies adaptively to maximize the target task performance.

Abstract

Transfer reinforcement learning (RL) aims at improving the learning efficiency of an agent by exploiting knowledge from other source agents trained on relevant tasks. However, it remains challenging to transfer knowledge between different environmental dynamics without having access to the source environments. In this work, we explore a new challenge in transfer RL, where only a set of source policies collected under diverse unknown dynamics is available for learning a target task efficiently. To address this problem, the proposed approach, MULTI-source POLicy AggRegation (MULTIPOLAR), comprises two key techniques. We learn to aggregate the actions provided by the source policies adaptively to maximize the target task performance. Meanwhile, we learn an auxiliary network that predicts residuals around the aggregated actions, which ensures the target policy's expressiveness even when some of the source policies perform poorly. We demonstrated the effectiveness of MULTIPOLAR through an extensive experimental evaluation across six simulated environments ranging from classic control problems to challenging robotics simulations, under both continuous and discrete action spaces. The demo videos and code are available on the project webpage: https://omron-sinicx.github.io/multipolar/.

Authors

Ryo Yonetani

2 papers

M. Barekatain

2 papers

Masashi Hamaya

1 papers

References44 items

Residual Reactive Navigation: Combining Classical and Learned Navigation Strategies For Deployment in Unknown Environments

Measuring Structural Similarities in Finite MDPs

Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization

Residual Policy Learning

Residual Reinforcement Learning for Robot Control

MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics

Published in

International Joint Conference on Artificial In...(2019)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Ryo Yonetani

2 papers

M. Barekatain

2 papers

Masashi Hamaya

1 papers

References44 items

Residual Reactive Navigation: Combining Classical and Learned Navigation Strategies For Deployment in Unknown Environments

Measuring Structural Similarities in Finite MDPs

Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization

Residual Policy Learning

Residual Reinforcement Learning for Robot Control

An Introduction to Deep Reinforcement Learning

Hardware Conditioned Policies for Multi-Robot Transfer Learning

Meta-Learning: A Survey

Policy Transfer with Strategy Optimization

Context-Aware Policy Reuse

Importance Weighted Transfer of Samples in Reinforcement Learning

Sim-to-Real: Learning Agile Locomotion For Quadruped Robots

Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning

Meta Reinforcement Learning with Latent Variable Gaussian Processes

Learning Robust Options

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Meta Learning Shared Hierarchies

An Optimal Online Method of Selecting Source Policies for Reinforcement Learning

Deep Reinforcement Learning that Matters

Proximal Policy Optimization Algorithms

Neural Combinatorial Optimization with Reinforcement Learning

Learning modular neural network policies for multi-task and multi-robot transfer

The Option-Critic Architecture

Measuring the Distance Between Finite Markov Decision Processes

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning

Attend, Adapt and Transfer: Attentive Deep Architecture for Adaptive Transfer from multiple sources in the same domain

Bayesian policy reuse

An automated measure of MDP similarity for transfer in reinforcement learning

Transfer Learning for Reinforcement Learning Domains: A Survey

Transfer of samples in batch reinforcement learning

Probabilistic policy reuse in a reinforcement learning agent

Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning

Introduction to Reinforcement Learning

An Introduction to the Bootstrap

A Deep Bayesian Policy Reuse Approach Against Non-Stationary Agents

Foundations and Trends ® in Machine Learning > Vol 7 > Issue 4-5 Ordering Info About Us Alerts Contact Help Log in Adaptation , Learning , and Optimization over Networks

TL;DR

Abstract

Authors

References44 items

Residual Reactive Navigation: Combining Classical and Learned Navigation Strategies For Deployment in Unknown Environments

Measuring Structural Similarities in Finite MDPs

Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization

Residual Policy Learning

Residual Reinforcement Learning for Robot Control

TL;DR

Abstract

Authors

References44 items

Residual Reactive Navigation: Combining Classical and Learned Navigation Strategies For Deployment in Unknown Environments

Measuring Structural Similarities in Finite MDPs

Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization

Residual Policy Learning

Residual Reinforcement Learning for Robot Control

An Introduction to Deep Reinforcement Learning

Hardware Conditioned Policies for Multi-Robot Transfer Learning

Meta-Learning: A Survey

Policy Transfer with Strategy Optimization

Context-Aware Policy Reuse

Importance Weighted Transfer of Samples in Reinforcement Learning

Sim-to-Real: Learning Agile Locomotion For Quadruped Robots

Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning

Meta Reinforcement Learning with Latent Variable Gaussian Processes

Learning Robust Options

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Meta Learning Shared Hierarchies

An Optimal Online Method of Selecting Source Policies for Reinforcement Learning

Deep Reinforcement Learning that Matters

Proximal Policy Optimization Algorithms

Neural Combinatorial Optimization with Reinforcement Learning

Learning modular neural network policies for multi-task and multi-robot transfer

The Option-Critic Architecture

Measuring the Distance Between Finite Markov Decision Processes

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning

Attend, Adapt and Transfer: Attentive Deep Architecture for Adaptive Transfer from multiple sources in the same domain

Bayesian policy reuse

An automated measure of MDP similarity for transfer in reinforcement learning

Transfer Learning for Reinforcement Learning Domains: A Survey

Transfer of samples in batch reinforcement learning

Probabilistic policy reuse in a reinforcement learning agent

Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning

Introduction to Reinforcement Learning

An Introduction to the Bootstrap

Stable Baselines

A Deep Bayesian Policy Reuse Approach Against Non-Stationary Agents

Ole Tange, 2018

Foundations and Trends ® in Machine Learning > Vol 7 > Issue 4-5 Ordering Info About Us Alerts Contact Help Log in Adaptation , Learning , and Optimization over Networks

The MIT Press

Recent Advances in Hierarchical Reinforcement Learning

Recent Advances in Hierarchical Reinforcement Learning

Accepted to the ICLR

GNU Parallel 2018

Field of Study

Venue Information

Name

Type

URL

Alternate Names