Counterfactual Multi-Agent Policy Gradients (2017-05-24T00:00:00.000000Z)

TL;DR

A new multi-agent actor-critic method called counterfactual multi- agent (COMA) policy gradients that uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies.

Abstract

Many real-world problems, such as network packet routing and the coordination of autonomous vehicles, are naturally modelled as cooperative multi-agent systems. There is a great need for new reinforcement learning methods that can efficiently learn decentralised policies for such systems. To this end, we propose a new multi-agent actor-critic method called counterfactual multi-agent (COMA) policy gradients. COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies. In addition, to address the challenges of multi-agent credit assignment, it uses a counterfactual baseline that marginalises out a single agent's action, while keeping the other agents' actions fixed. COMA also uses a critic representation that allows the counterfactual baseline to be computed efficiently in a single forward pass. We evaluate COMA in the testbed of StarCraft unit micromanagement, using a decentralised variant with significant partial observability. COMA significantly improves average performance over other multi-agent actor-critic methods in this setting, and the best performing agents are competitive with state-of-the-art centralised controllers that get access to the full state.

Authors

Gregory Farquhar

5 papers

Jakob N. Foerster

8 papers

Shimon Whiteson

13 papers

TL;DR

Abstract

Authors

References52 items

Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

Cooperative Multi-agent Control Using Deep Reinforcement Learning

Multiagent Bidirectionally-Coordinated Nets for Learning to Play StarCraft Combat Games

Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games

Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning

Deep Decentralized Multi-task Multi-Agent RL under Partial Observability

Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability

Emergence of Grounded Compositional Language in Multi-Agent Populations

Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning

Multi-agent Reinforcement Learning in Sequential Social Dilemmas

Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence

Multi-Agent Cooperation and the Emergence of (Natural) Language

Sample Efficient Actor-Critic with Experience Replay

TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games

Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks

Learning Multiagent Communication with Backpropagation

Learning to Communicate with Deep Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning as a rehearsal for decentralized planning

Multiagent cooperation and competition with deep reinforcement learning

Deep Recurrent Q-Learning for Partially Observable MDPs

High-Dimensional Continuous Control Using Generalized Advantage Estimation

Approximating Difference Evaluations with Local Information

A Multi-Agent Framework for Packet Routing in Wireless Sensor Networks

Human-level control through deep reinforcement learning

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

Empirically Evaluating Multiagent Learning Algorithms

An Overview of Recent Progress in the Study of Distributed Multi-Agent Coordination

Modeling difference rewards for multiagent learning

Reinforcement learning in feedback control

Optimal and Approximate Q-value Functions for Decentralized POMDPs

A Comprehensive Survey of Multiagent Reinforcement Learning

Distributed agent-based air traffic flow management

Software agent-based applications, platforms and development kits

Multi-agent framework for third party logistics in E-commerce

All learning is Local: Multi-agent Learning in Global Reward Games

The Optimal Reward Baseline for Gradient-Based Reinforcement Learning

Optimal Payoff Functions for Members of Collectives

Policy Gradient Methods for Reinforcement Learning with Function Approximation

An Analysis of Actor/Critic Algorithms Using Eligibility Traces: Reinforcement Learning with Imperfect Value Function

Long Short-Term Memory

Learning from delayed rewards

Learning to predict by the methods of temporal differences

tuning the skip-frame in StarCraft can improve absolute performance

Torch7: A Matlab-like Environment for Machine Learning

Multiagent Systems: Introduction to Noncooperative Game Theory: Games in Normal Form

The Packet-World: A testbed for investigating situated multiagent systems

Multiagent Reinforcement Learning for Multi-Robot Systems: A Survey

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Multi Agent Reinforcement Learning Independent vs Cooperative Agents

Actor-Critic Algorithms

On-line Q-learning using connectionist systems

Field of Study

Venue Information

Name

Type

URL

Alternate Names