Learning Synthetic Environments for Reinforcement Learning with Evolution Strategies (2021-01-24T00:00:00.000000Z)

TL;DR

This work explores learning agent-agnostic synthetic environments (SEs) for Reinforcement Learning as a bi-level optimization problem and represents an SE as a neural network by using Natural Evolution Strategies and a population of SE parameter vectors.

Abstract

This work explores learning agent-agnostic synthetic environments (SEs) for Reinforcement Learning. SEs act as a proxy for target environments and allow agents to be trained more efficiently than when directly trained on the target environment. We formulate this as a bi-level optimization problem and represent an SE as a neural network. By using Natural Evolution Strategies and a population of SE parameter vectors, we train agents in the inner loop on evolving SEs while in the outer loop we use the performance on the target task as a score for meta-updating the SE population. We show empirically that our method is capable of learning SEs for two discrete-action-space tasks (CartPole-v0 and Acrobot-v1) that allow us to train agents more robustly and with up to 60% fewer steps. Not only do we show in experiments with 4000 evaluations that the SEs are robust against hyperparameter changes such as the learning rate, batch sizes and network sizes, we also show that SEs trained with DDQN agents transfer in limited ways to a discrete-action-space version of TD3 and very well to Dueling DDQN.

Authors

F. Hutter

14 papers

Fábio Ferreira

2 papers

Thomas Nierhoff

1 papers

Learning Synthetic Environments for Reinforcement Learning with Evolution Strategies

TL;DR

Abstract

Authors

References20 items

Fully Differentiable Procedural Content Generation through Generative Playing Networks

Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data

Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions

Understanding and correcting pathologies in the training of learned optimizers

BOHB: Robust and Efficient Hyperparameter Optimization at Scale

On Learning Intrinsic Rewards for Policy Gradient Methods

World Models

Addressing Function Approximation Error in Actor-Critic Methods

Active Learning for Convolutional Neural Networks: A Core-Set Approach

Evolution Strategies as a Scalable Alternative to Reinforcement Learning

Categorical Reparameterization with Gumbel-Softmax

OpenAI Gym

Dueling Network Architectures for Deep Reinforcement Learning

Deep Reinforcement Learning with Double Q-Learning

Mirrored Sampling and Sequential Selection for Evolution Strategies

Natural Evolution Strategies

Policy Gradient Methods for Reinforcement Learning with Function Approximation

Learning from delayed rewards

Automated Machine Learning - Methods, Systems, Challenges

Evolutionsstrategie : Optimierung technischer Systeme nach Prinzipien der biologischen Evolution

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names