Evolution Strategies as a Scalable Alternative to Reinforcement Learning (2017-03-10T00:00:00.000000Z)

TL;DR

This work explores the use of Evolution Strategies (ES), a class of black box optimization algorithms, as an alternative to popular MDP-based RL techniques such as Q-learning and Policy Gradients, and highlights several advantages of ES as a blackbox optimization technique.

Abstract

We explore the use of Evolution Strategies (ES), a class of black box optimization algorithms, as an alternative to popular MDP-based RL techniques such as Q-learning and Policy Gradients. Experiments on MuJoCo and Atari show that ES is a viable solution strategy that scales extremely well with the number of CPUs available: By using a novel communication strategy based on common random numbers, our ES implementation only needs to communicate scalars, making it possible to scale to over a thousand parallel workers. This allows us to solve 3D humanoid walking in 10 minutes and obtain competitive results on most Atari games after one hour of training. In addition, we highlight several advantages of ES as a black box optimization technique: it is invariant to action frequency and delayed rewards, tolerant of extremely long horizons, and does not need temporal discounting or value function approximation.

Authors

I. Sutskever

26 papers

Tim Salimans

8 papers

Xi Chen

4 papers

TL;DR

Abstract

Authors

References47 items

In-datacenter performance analysis of a tensor processing unit

RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning

Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks

A Wavelet-based Encoding for Neuroevolution

Improved Techniques for Training GANs

Deep Learning without Poor Local Minima

Benchmarking Deep Reinforcement Learning for Continuous Control

Asynchronous Methods for Deep Reinforcement Learning

Mastering the game of Go with deep neural networks and tree search

Random Gradient-Free Minimization of Convex Functions

High-Dimensional Continuous Control Using Generalized Advantage Estimation

End-to-End Training of Deep Visuomotor Policies

Human-level control through deep reinforcement learning

Trust Region Policy Optimization

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Neuroevolution in Games: State of the Art and Open Challenges

A Neuroevolution Approach to General Atari Game Playing

Optimal Rates for Zero-Order Convex Optimization: The Power of Two Function Evaluations

Evolving large-scale neural networks for vision-based reinforcement learning

MuJoCo: A physics engine for model-based control

Policy Improvement Methods: Between Black-Box Optimization and Episodic Reinforcement Learning

Generalized compressed network search

Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems

High dimensions and heavy tails for natural evolution strategies

Mirrored Sampling and Sequential Selection for Evolution Strategies

A Natural Evolution Strategy for Multi-objective Optimization

Evolving neural networks in compressed weight space

Exponential natural evolution strategies

2010 Special Issue: Parameter-exploring policy gradients

Efficient natural evolution strategies

Stochastic search using the natural gradient

A Hypercube-Based Encoding for Evolving Large-Scale Neural Networks

Natural Evolution Strategies

Training Recurrent Networks by Evolino

Completely Derandomized Self-Adaptation in Evolution Strategies

Direct Policy Search and Uncertain Policy Evaluation

Reinforcement Learning with Hierarchies of Machines

Simple statistical gradient-following algorithms for connectionist reinforcement learning

Multivariate stochastic approximation using a simultaneous perturbation gradient approximation

Antithetic acceleration of Monte Carlo integration in Bayesian inference

The data efﬁciency of evolution strategies was surprisingly good: we were able to match the ﬁnal performance of A3C

Frame Skip Is a Powerful Parameter for Learning to Play Atari

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Autonomous Inverted Helicopter Flight via Reinforcement Learning

Numerische Optimierung von Computermodellen mittels der Evo-lutionsstrategie

Evolutionsstrategie : Optimierung technischer Systeme nach Prinzipien der biologischen Evolution

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names