A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation

Published in

Neural Information Processing Systems(2019)

External Links:

Generate Graph

TL;DR

A generalized version of the Bellman equation is proposed to learn a single parametric representation for optimal policies over the space of all possible preferences in MORL, with the goal of enabling few-shot adaptation to new tasks.

Abstract

We introduce a new algorithm for multi-objective reinforcement learning (MORL) with linear preferences, with the goal of enabling few-shot adaptation to new tasks. In MORL, the aim is to learn policies over multiple competing objectives whose relative importance (preferences) is unknown to the agent. While this alleviates dependence on scalar reward design, the expected return of a policy can change significantly with varying preferences, making it challenging to learn a single model to produce optimal policies under different preference conditions. We propose a generalized version of the Bellman equation to learn a single parametric representation for optimal policies over the space of all possible preferences. After an initial learning phase, our agent can execute the optimal policy under any given preference, or automatically infer an underlying preference with very few samples. Experiments across four different domains demonstrate the effectiveness of our approach.

Authors

Xingyuan Sun

3 papers

Runzhe Yang

2 papers

Karthik Narasimhan

7 papers

References52 items

Dynamic Programming

Dynamic Weights in Multi-Objective Deep Reinforcement Learning

Manifold-based multi-objective policy search with sample reuse

A Distributional Perspective on Reinforcement Learning

Hindsight Experience Replay

A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation

Published in

Neural Information Processing Systems(2019)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Xingyuan Sun

3 papers

Runzhe Yang

2 papers

Karthik Narasimhan

7 papers

References52 items

Dynamic Programming

Dynamic Weights in Multi-Objective Deep Reinforcement Learning

Manifold-based multi-objective policy search with sample reuse

A Distributional Perspective on Reinforcement Learning

Hindsight Experience Replay

PyDial: A Multi-domain Statistical Dialogue System Toolkit

Reward-Balancing for Statistical Spoken Dialogue Systems using Multi-objective Reinforcement Learning

Discrete Sequential Prediction of Continuous Actions for Deep RL

Multi-Objective Deep Reinforcement Learning

Regular Policies in Abstract Dynamic Programming

Concrete Problems in AI Safety

Generative Adversarial Imitation Learning

Continuous Deep Q-Learning with Model-based Acceleration

Asynchronous Methods for Deep Reinforcement Learning

Prioritized Experience Replay

Deep Reinforcement Learning with Double Q-Learning

Continuous control with deep reinforcement learning

Multiobjective Reinforcement Learning: A Comprehensive Overview

Human-level control through deep reinforcement learning

Multi-objective reinforcement learning for acquiring all Pareto optimal policies simultaneously - Method of determining scalarization weights

Multi-Objective Reinforcement Learning with Continuous Pareto Frontier Approximation

A Survey of Multi-Objective Sequential Decision-Making

Abstract dynamic programming

Scalarized multi-objective reinforcement learning: Novel design techniques

Tree-based Fitted Q-iteration for Multi-Objective Markov Decision problems

Empirical evaluation methods for multiobjective reinforcement learning algorithms

Multi-objective fitted Q-iteration: Pareto frontier approximation in one single run

Sequential Approximate Multiobjective Optimization Using Computational Intelligence

The Hidden Agenda User Simulation Model

Learning all optimal policies with multiple criteria

Managing Power Consumption and Performance of Computing Systems Using Reinforcement Learning

Parallel reinforcement learning for weighted multi-criteria model with adaptive margin

Multi-objective optimization using genetic algorithms: A tutorial

Adaptive weighted sum method for multiobjective optimization: a new method for Pareto front generation

Dynamic preferences in multi-criteria reinforcement learning

On min-norm and min-max methods of multi-objective optimization

Apprenticeship learning via inverse reinforcement learning

A POMDP formulation of preference elicitation problems

Preference elicitation in combinatorial auctions

Learning an Agent's Utility Function by Observing Behavior

An Introduction to Metric Spaces and Fixed Point Theory

The Steering Approach for Multi-Criteria Reinforcement Learning

Algorithms for Inverse Reinforcement Learning

Modern homotopy methods in optimization

Super Mario Bros for OpenAI Gym

Visualizing Data using t-SNE

Technical Note: Q-Learning

Survey of Preference Elicitation Methods

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Linear Operator Theory in Engineering and Science

Task-Oriented Dialog Policy Learning (Dialog) A modiﬁed task-oriented dialog system in the restaurant reservation domain based on PyDial [36]

Field of Study

Computer Science

Venue Information

Name

Neural Information Processing Systems

Type

conference

URL

http://neurips.cc/

Alternate Names

Neural Inf Process Syst
NeurIPS
NIPS

TL;DR

Abstract

Authors

References52 items

Dynamic Programming

Dynamic Weights in Multi-Objective Deep Reinforcement Learning

Manifold-based multi-objective policy search with sample reuse

A Distributional Perspective on Reinforcement Learning

Hindsight Experience Replay

TL;DR

Abstract

Authors

References52 items

Dynamic Programming

Dynamic Weights in Multi-Objective Deep Reinforcement Learning

Manifold-based multi-objective policy search with sample reuse

A Distributional Perspective on Reinforcement Learning

Hindsight Experience Replay

PyDial: A Multi-domain Statistical Dialogue System Toolkit

Reward-Balancing for Statistical Spoken Dialogue Systems using Multi-objective Reinforcement Learning

Discrete Sequential Prediction of Continuous Actions for Deep RL

Multi-Objective Deep Reinforcement Learning

Regular Policies in Abstract Dynamic Programming

Concrete Problems in AI Safety

Generative Adversarial Imitation Learning

Continuous Deep Q-Learning with Model-based Acceleration

Asynchronous Methods for Deep Reinforcement Learning

Prioritized Experience Replay

Deep Reinforcement Learning with Double Q-Learning

Continuous control with deep reinforcement learning

Multiobjective Reinforcement Learning: A Comprehensive Overview

Human-level control through deep reinforcement learning

Multi-objective reinforcement learning for acquiring all Pareto optimal policies simultaneously - Method of determining scalarization weights

Multi-Objective Reinforcement Learning with Continuous Pareto Frontier Approximation

A Survey of Multi-Objective Sequential Decision-Making

Abstract dynamic programming

Scalarized multi-objective reinforcement learning: Novel design techniques

Tree-based Fitted Q-iteration for Multi-Objective Markov Decision problems

Empirical evaluation methods for multiobjective reinforcement learning algorithms

Multi-objective fitted Q-iteration: Pareto frontier approximation in one single run

Sequential Approximate Multiobjective Optimization Using Computational Intelligence

The Hidden Agenda User Simulation Model

Learning all optimal policies with multiple criteria

Managing Power Consumption and Performance of Computing Systems Using Reinforcement Learning

Parallel reinforcement learning for weighted multi-criteria model with adaptive margin

Multi-objective optimization using genetic algorithms: A tutorial

Adaptive weighted sum method for multiobjective optimization: a new method for Pareto front generation

Dynamic preferences in multi-criteria reinforcement learning

On min-norm and min-max methods of multi-objective optimization

Apprenticeship learning via inverse reinforcement learning

A POMDP formulation of preference elicitation problems

Preference elicitation in combinatorial auctions

Learning an Agent's Utility Function by Observing Behavior

An Introduction to Metric Spaces and Fixed Point Theory

The Steering Approach for Multi-Criteria Reinforcement Learning

Algorithms for Inverse Reinforcement Learning

Modern homotopy methods in optimization

Super Mario Bros for OpenAI Gym

Visualizing Data using t-SNE

Technical Note: Q-Learning

Survey of Preference Elicitation Methods

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Linear Operator Theory in Engineering and Science

Task-Oriented Dialog Policy Learning (Dialog) A modiﬁed task-oriented dialog system in the restaurant reservation domain based on PyDial [36]

expected returns

Field of Study

Venue Information

Name

Type

URL

Alternate Names