Emergent Complexity via Multi-Agent Competition (2017-10-10T00:00:00.000000Z)

TL;DR

This work introduces several competitive multi-agent environments where agents compete in a 3D world with simulated physics and points out that such environments come with a natural curriculum, because for any skill level, an environment full of agents of this level will have the right level of difficulty.

Abstract

Reinforcement learning algorithms can train agents that solve problems in complex, interesting environments. Normally, the complexity of the trained agent is closely related to the complexity of the environment. This suggests that a highly capable agent requires a complex environment for training. In this paper, we point out that a competitive multi-agent environment trained with self-play can produce behaviors that are far more complex than the environment itself. We also point out that such environments come with a natural curriculum, because for any skill level, an environment full of agents of this level will have the right level of difficulty. This work introduces several competitive multi-agent environments where agents compete in a 3D world with simulated physics. The trained agents learn a wide variety of complex and interesting skills, even though the environment themselves are relatively simple. The skills include behaviors such as running, blocking, ducking, tackling, fooling opponents, kicking, and defending using both arms and legs. A highlight of the learned behaviors can be found here: this https URL

Authors

I. Sutskever

26 papers

J. Pachocki

4 papers

Szymon Sidor

3 papers

TL;DR

Abstract

Authors

References36 items

Learning with Opponent-Learning Awareness

Proximal Policy Optimization Algorithms

Emergence of Locomotion Behaviours in Rich Environments

Hindsight Experience Replay

Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

Counterfactual Multi-Agent Policy Gradients

Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play

Supervision via competition: Robot adversaries for learning tasks

Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm

Opponent Modeling in Deep Reinforcement Learning

Benchmarking Deep Reinforcement Learning for Continuous Control

Deep Reinforcement Learning from Self-Play in Imperfect-Information Games

Asynchronous Methods for Deep Reinforcement Learning

Mastering the game of Go with deep neural networks and tree search

Multiagent cooperation and competition with deep reinforcement learning

Continuous control with deep reinforcement learning

High-Dimensional Continuous Control Using Generalized Advantage Estimation

Human-level control through deep reinforcement learning

Trust Region Policy Optimization

Adam: A Method for Stochastic Optimization

MuJoCo: A physics engine for model-based control

Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems

Competitive Coevolution through Evolutionary Complexification

Character animation in two-player adversarial games

A Comprehensive Survey of Multiagent Reinforcement Learning

Cooperative Multi-Agent Learning: The State of the Art

Multitask Learning

Temporal Difference Learning and TD-Gammon

Evolving virtual creatures

Markov Games as a Framework for Multi-Agent Reinforcement Learning

GENERATIVE ADVERSARIAL NETS

OpenAI

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Multi Agent Reinforcement Learning Independent vs Cooperative Agents

You Shall not Pass For the agent whose goal is to reach the other side, the reward is same as for run-to-goal. For the blocking agent, the reward for humanoid is c h (s, a) + C alive + |x − g| where

Run to Goal For humanoids, reward is r h (s, a) − |x − g| where x − g is the l 1 distance of the agent from the goal g along the x-axis. For ant, reward is similar r q (s, a) − |x − g|

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names