ACE: An Actor Ensemble Algorithm for Continuous Control with Tree Search (2018-11-06T00:00:00.000000Z)

TL;DR

This paper proposes an actor ensemble algorithm, named ACE, for continuous control with a deterministic policy in reinforcement learning, and forms ACE in the option framework by extending the option-critic architecture with deterministic intra-option policies, revealing a relationship between ensemble and options.

Abstract

In this paper, we propose an actor ensemble algorithm, named ACE, for continuous control with a deterministic policy in reinforcement learning. In ACE, we use actor ensemble (i.e., multiple actors) to search the global maxima of the critic. Besides the ensemble perspective, we also formulate ACE in the option framework by extending the option-critic architecture with deterministic intra-option policies, revealing a relationship between ensemble and options. Furthermore, we perform a look-ahead tree search with those actors and a learned value prediction model, resulting in a refined value estimation. We demonstrate a significant performance boost of ACE over DDPG and its variants in challenging physical robot simulators.

Authors

Shangtong Zhang

2 papers

Hao Chen

2 papers

Hengshuai Yao

2 papers

TL;DR

Abstract

Authors

References33 items

Distributed Distributional Deterministic Policy Gradients

DeepMind Control Suite

Learning to Run with Actor-Critic Ensemble

Learnings Options End-to-End for Continuous Action Tasks

TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning

Imagination-Augmented Agents for Deep Reinforcement Learning

Value Prediction Network

Monte Carlo Tree Search in Continuous Action Spaces with Execution Uncertainty

Continuous Deep Q-Learning with Model-based Acceleration

Deep Exploration via Bootstrapped DQN

Mastering the game of Go with deep neural networks and tree search

Continuous control with deep reinforcement learning

Planning in Discrete and Continuous Markov Decision Processes by Probabilistic Programming

Human-level control through deep reinforcement learning

Deterministic Policy Gradient Algorithms

A Survey of Monte Carlo Tree Search Methods

Sample-Based Planning for Continuous Action Markov Decision Processes

Ensembles of Neural Networks for Robust Reinforcement Learning

Rectified Linear Units Improve Restricted Boltzmann Machines

AN ANALYSIS OF UCT IN MULTI-PLAYER GAMES

Ensemble Algorithms in Reinforcement Learning

Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search

Policy Gradient Methods for Reinforcement Learning with Function Approximation

Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning

Temporal Difference Learning and TD-Gammon

Q-learning

Self-improving reactive agents based on reinforcement learning, planning and teaching

The Solution for the Branching Factor of the Alpha-Beta Pruning Algorithm

The optioncritic architecture

Multi-Step Dyna Planning for Policy Evaluation and Control

Off‐Policy Actor‐Criticアルゴリズムによる強化学習

Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming

On the theory of brownian motion

Field of Study

Venue Information

Name

Type

URL

Alternate Names