A Distributional View on Multi-Objective Policy Optimization

Published in

International Conference on Machine Learning(2020)

External Links:

Generate Graph

TL;DR

This paper proposes a novel algorithm for multi-objective reinforcement learning that enables setting desired preferences for objectives in a scale-invariant way, and uses supervised learning to fit a parametric policy to a combination of these distributions.

Abstract

Many real-world problems require trading off multiple competing objectives. However, these objectives are often in different units and/or scales, which can make it challenging for practitioners to express numerical preferences over objectives in their native units. In this paper we propose a novel algorithm for multi-objective reinforcement learning that enables setting desired preferences for objectives in a scale-invariant way. We propose to learn an action distribution for each objective, and we use supervised learning to fit a parametric policy to a combination of these distributions. We demonstrate the effectiveness of our approach on challenging high-dimensional real and simulated robotics tasks, and show that setting different preferences in our framework allows us to trace out the space of nondominated solutions.

Authors

H. F. Song

4 papers

N. Heess

9 papers

Martin A. Riedmiller

7 papers

References70 items

CoMic: Co-Training and Mimicry for Reusable Skills

Relationship Explainable Multi-objective Optimization Via Vector Value Function Based Reinforcement Learning

V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control

Using Logical Specifications of Objectives in Multi-Objective Reinforcement Learning

A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation

A Distributional View on Multi-Objective Policy Optimization

Published in

International Conference on Machine Learning(2020)

External Links:

Generate Graph

TL;DR

Abstract

Authors

H. F. Song

4 papers

N. Heess

9 papers

Martin A. Riedmiller

7 papers

References70 items

CoMic: Co-Training and Mimicry for Reusable Skills

Relationship Explainable Multi-objective Optimization Via Vector Value Function Based Reinforcement Learning

V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control

Using Logical Specifications of Objectives in Multi-Objective Reinforcement Learning

A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation

R. Hadsell

10 papers

A. Abdolmaleki

5 papers

Sandy H. Huang

1 papers

Leonard Hasenclever

2 papers

Martina Zambelli

1 papers

Regularized Hierarchical Policies for Compositional Transfer in Robotics

TossingBot: Learning to Throw Arbitrary Objects With Residual Physics

Batch Policy Learning under Constraints

Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning

Value constrained model-free continuous control

Relative Entropy Regularized Policy Iteration

Meta-Learning for Multi-objective Reinforcement Learning

Physics-based motion capture imitation with deep reinforcement learning

Hierarchical visuomotor control of humanoids

Dynamic Weights in Multi-Objective Deep Reinforcement Learning

Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning

Multi-task Deep Reinforcement Learning with PopArt

A Lyapunov-based Approach to Safe Reinforcement Learning

Sim-to-Real: Learning Agile Locomotion For Quadruped Robots

Learning by Playing - Solving Sparse Reward Tasks from Scratch

Maximum a Posteriori Policy Optimisation

Distributed Distributional Deterministic Policy Gradients

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

DeepMind Control Suite

On the effect of normalization in MOEA/D for multi-objective and many-objective optimization

Proximal Policy Optimization Algorithms

Distral: Robust multitask reinforcement learning

Hybrid Reward Architecture for Reinforcement Learning

Constrained Policy Optimization

Multi-Objective Deep Reinforcement Learning

Concrete Problems in AI Safety

Safe and Efficient Off-Policy Reinforcement Learning

Learning values across many orders of magnitude

Mastering the game of Go with deep neural networks and tree search

Learning Continuous Control Policies by Stochastic Value Gradients

End-to-End Training of Deep Visuomotor Policies

Multiobjective Reinforcement Learning: A Comprehensive Overview

Human-level control through deep reinforcement learning

Trust Region Policy Optimization

Multi-Objective MDPs with Conditional Lexicographic Reward Preferences

Adam: A Method for Stochastic Optimization

New perspectives on the natural gradient method

Policy gradient approaches for multi-objective sequential decision making

Multi-Objective Reinforcement Learning with Continuous Pareto Frontier Approximation

Linear support for multi-objective coordination graphs

A Survey of Multi-Objective Sequential Decision-Making

Scalarized multi-objective reinforcement learning: Novel design techniques

MuJoCo: A physics engine for model-based control

Empirical evaluation methods for multiobjective reinforcement learning algorithms

Learning all optimal policies with multiple criteria

Adaptive weighted sum method for multiobjective optimization: a new method for Pareto front generation

Function-transformation methods for multi-objective optimization

Dynamic preferences in multi-criteria reinforcement learning

Q-Decomposition for Reinforcement Learning Agents

Learning from delayed rewards

Learning to predict by the methods of temporal differences

CVXOPT: A Python package for convex optimization, version

Humanoid Mocap The motion capture tracking task used in this paper was developed as part of a concurrent submission (Anonymous

CoMic: Complementary Task Learning & Mimicry for Reusable Skills

Shadow Robot Company

Pareto-DQN: Approximating the Pareto front in complex multi-objective decision problems

A robust proportion-preserving composite objective function for scale-invariant multi-objective optimization

e-PAL: An Active Learning Approach to the Multi-Objective Optimization Problem

Multi-objective reinforcement learning using sets of pareto dominating policies

The gesture recognition toolkit

DEAP: evolutionary algorithms made easy

Treasures are labeled with their respective values. The agent can move around freely in the white

Normalization and Other Topics in MultiObjective Optimization

Multi-criteria Reinforcement Learning

Field of Study

Computer ScienceMathematics

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

International Conference on Machine Learning

Type

conference

URL

https://icml.cc/

Alternate Names

ICML
Int Conf Mach Learn

TL;DR

Abstract

Authors

References70 items

CoMic: Co-Training and Mimicry for Reusable Skills

Relationship Explainable Multi-objective Optimization Via Vector Value Function Based Reinforcement Learning

V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control

Using Logical Specifications of Objectives in Multi-Objective Reinforcement Learning

A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation

TL;DR

Abstract

Authors

References70 items

CoMic: Co-Training and Mimicry for Reusable Skills

Relationship Explainable Multi-objective Optimization Via Vector Value Function Based Reinforcement Learning

V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control

Using Logical Specifications of Objectives in Multi-Objective Reinforcement Learning

A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation

Regularized Hierarchical Policies for Compositional Transfer in Robotics

TossingBot: Learning to Throw Arbitrary Objects With Residual Physics

Batch Policy Learning under Constraints

Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning

Value constrained model-free continuous control

Relative Entropy Regularized Policy Iteration

Meta-Learning for Multi-objective Reinforcement Learning

Physics-based motion capture imitation with deep reinforcement learning

Hierarchical visuomotor control of humanoids

Dynamic Weights in Multi-Objective Deep Reinforcement Learning

Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning

Multi-task Deep Reinforcement Learning with PopArt

A Lyapunov-based Approach to Safe Reinforcement Learning

Sim-to-Real: Learning Agile Locomotion For Quadruped Robots

DeepMimic

Learning by Playing - Solving Sparse Reward Tasks from Scratch

Maximum a Posteriori Policy Optimisation

Distributed Distributional Deterministic Policy Gradients

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

DeepMind Control Suite

On the effect of normalization in MOEA/D for multi-objective and many-objective optimization

Proximal Policy Optimization Algorithms

Distral: Robust multitask reinforcement learning

Hybrid Reward Architecture for Reinforcement Learning

Constrained Policy Optimization

Multi-Objective Deep Reinforcement Learning

Concrete Problems in AI Safety

Safe and Efficient Off-Policy Reinforcement Learning

Learning values across many orders of magnitude

Mastering the game of Go with deep neural networks and tree search

Learning Continuous Control Policies by Stochastic Value Gradients

End-to-End Training of Deep Visuomotor Policies

Multiobjective Reinforcement Learning: A Comprehensive Overview

Human-level control through deep reinforcement learning

Trust Region Policy Optimization

Multi-Objective MDPs with Conditional Lexicographic Reward Preferences

Adam: A Method for Stochastic Optimization

New perspectives on the natural gradient method

Policy gradient approaches for multi-objective sequential decision making

Multi-Objective Reinforcement Learning with Continuous Pareto Frontier Approximation

Linear support for multi-objective coordination graphs

A Survey of Multi-Objective Sequential Decision-Making

Scalarized multi-objective reinforcement learning: Novel design techniques

MuJoCo: A physics engine for model-based control

Empirical evaluation methods for multiobjective reinforcement learning algorithms

Learning all optimal policies with multiple criteria

Adaptive weighted sum method for multiobjective optimization: a new method for Pareto front generation

Function-transformation methods for multi-objective optimization

Dynamic preferences in multi-criteria reinforcement learning

Q-Decomposition for Reinforcement Learning Agents

Learning from delayed rewards

Learning to predict by the methods of temporal differences

CVXOPT: A Python package for convex optimization, version

Humanoid Mocap The motion capture tracking task used in this paper was developed as part of a concurrent submission (Anonymous

CoMic: Complementary Task Learning & Mimicry for Reusable Skills

Shadow Robot Company

Pareto-DQN: Approximating the Pareto front in complex multi-objective decision problems

A robust proportion-preserving composite objective function for scale-invariant multi-objective optimization

e-PAL: An Active Learning Approach to the Multi-Objective Optimization Problem

Multi-objective reinforcement learning using sets of pareto dominating policies

The gesture recognition toolkit

DEAP: evolutionary algorithms made easy

Normalization and Other Topics in MultiObjective Optimization