Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning (2017-02-28T00:00:00.000000Z)

TL;DR

Two methods using a multi-agent variant of importance sampling to naturally decay obsolete data and conditioning each agent's value function on a fingerprint that disambiguates the age of the data sampled from the replay memory enable the successful combination of experience replay with multi- agent RL.

Abstract

Many real-world problems, such as network packet routing and urban traffic control, are naturally modeled as multi-agent reinforcement learning (RL) problems. However, existing multi-agent RL methods typically scale poorly in the problem size. Therefore, a key challenge is to translate the success of deep learning on single-agent RL to the multi-agent setting. A major stumbling block is that independent Q-learning, the most popular multi-agent RL method, introduces nonstationarity that makes it incompatible with the experience replay memory on which deep Q-learning relies. This paper proposes two methods that address this problem: 1) using a multi-agent variant of importance sampling to naturally decay obsolete data and 2) conditioning each agent's value function on a fingerprint that disambiguates the age of the data sampled from the replay memory. Results on a challenging decentralised variant of StarCraft unit micro-management confirm that these methods enable the successful combination of experience replay with multi-agent RL.

Authors

Pushmeet Kohli

16 papers

Gregory Farquhar

5 papers

Jakob N. Foerster

8 papers

TL;DR

Abstract

Authors

References40 items

GROUNDS OF MODERN MODELS AND SYSTEMS OF ORGANIZATIONAL CREATIVITY SUPPORT

OFFER: Off-Environment Reinforcement Learning

Multi-agent Reinforcement Learning in Sequential Social Dilemmas

Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence

Sample Efficient Actor-Critic with Experience Replay

TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games

Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks

Opponent Modeling in Deep Reinforcement Learning

Learning Multiagent Communication with Backpropagation

Learning to Communicate with Deep Multi-Agent Reinforcement Learning

Multiagent cooperation and competition with deep reinforcement learning

Prioritized Experience Replay

Deep Recurrent Q-Learning for Partially Observable MDPs

A Multi-Agent Framework for Packet Routing in Wireless Sensor Networks

Human-level control through deep reinforcement learning

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Empirically Evaluating Multiagent Learning Algorithms

Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems

Multiagent Reinforcement Learning for Urban Traffic Control Using Coordination Graphs

A Comprehensive Survey of Multiagent Reinforcement Learning

Collaborative Multiagent Reinforcement Learning by Payoff Propagation

Dealing with non-stationary environments using context detection

Software agent-based applications, platforms and development kits

Monte Carlo Statistical Methods

Extending Q-Learning to General Adaptive Multi-Agent Systems

AWESOME: A general multiagent learning algorithm that converges in self-play and learns a best response against stationary opponents

Hierarchical multi-agent reinforcement learning

An Algorithm for Distributed Reinforcement Learning in Cooperative Multi-Agent Systems

Introduction to Reinforcement Learning

Long Short-Term Memory

Using communication to reduce locality in distributed multiagent learning

Learning from delayed rewards

Half Field Offense: An Environment for Multiagent Learning and Ad Hoc Teamwork

Coordinated Deep Reinforcement Learners for Traffic Light Control

Torch7: A Matlab-like Environment for Machine Learning

Multiagent Systems: Introduction to Noncooperative Game Theory: Games in Normal Form

The Packet-World: A testbed for investigating situated multiagent systems

Multiagent Reinforcement Learning for Multi-Robot Systems: A Survey

Multi Agent Reinforcement Learning Independent vs Cooperative Agents

Reinforcement Learning: An Introduction

Field of Study

Venue Information

Name

Type

URL

Alternate Names