QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018-03-30T00:00:00.000000Z)

TL;DR

QMIX employs a network that estimates joint action-values as a complex non-linear combination of per-agent values that condition only on local observations, and structurally enforce that the joint-action value is monotonic in the per- agent values, which allows tractable maximisation of the jointaction-value in off-policy learning.

Abstract

In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a network that estimates joint action-values as a complex non-linear combination of per-agent values that condition only on local observations. We structurally enforce that the joint-action value is monotonic in the per-agent values, which allows tractable maximisation of the joint action-value in off-policy learning, and guarantees consistency between the centralised and decentralised policies. We evaluate QMIX on a challenging set of StarCraft II micromanagement tasks, and show that QMIX significantly outperforms existing value-based multi-agent reinforcement learning methods.

Authors

Tabish Rashid

5 papers

Mikayel Samvelyan

7 papers

C. S. D. Witt

6 papers

TL;DR

Abstract

Authors

References77 items

Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery

Grandmaster level in StarCraft II using multi-agent reinforcement learning

Learning Nearly Decomposable Value Functions Via Communication Minimization

Deep Coordination Graphs

From Few to More: Large-scale Dynamic Multiagent Curriculum Learning

Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control

rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch

Value Function Transfer for Deep Multi-Agent Reinforcement Learning Based on N-Step Returns

Stratospheric Aerosol Injection as a Deep Reinforcement Learning Problem

QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning

Learning Efficient Communication in Cooperative Multi-Agent Environment

Multiagent Learning and Coordination with Clustered Deep Q-Network

Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces

The Representational Capacity of Action-Value Networks for Multi-Agent Reinforcement Learning

The StarCraft Multi-Agent Challenge

CESMA: Centralized Expert Supervises Multi-Agents

The Hanabi Challenge: A New Frontier for AI Research

Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning

Deep Multi-Agent Reinforcement Learning with Relevance Graphs

Multi-Agent Common Knowledge Reinforcement Learning

Actor-Attention-Critic for Multi-Agent Reinforcement Learning

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

Factorized Q-learning for large-scale multi-agent systems

Deep Reinforcement Learning for Swarm Systems

Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward

Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research

Machine Theory of Mind

Diversity is All You Need: Learning Skills without a Reward Function

Mean Field Multi-Agent Reinforcement Learning

RLlib: Abstractions for Distributed Reinforcement Learning

MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence

Guided Deep Reinforcement Learning for Swarm Systems

StarCraft II: A New Challenge for Reinforcement Learning

A Distributional Perspective on Reinforcement Learning

Lenient Multi-Agent Deep Reinforcement Learning

Value-Decomposition Networks For Cooperative Multi-Agent Learning

Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

Counterfactual Multi-Agent Policy Gradients

Cooperative Multi-agent Control Using Deep Reinforcement Learning

Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games

Deep Decentralized Multi-task Multi-Agent RL under Partial Observability

Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability

Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning

Multi-agent Reinforcement Learning in Sequential Social Dilemmas

Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence

TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games

HyperNetworks

Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks

A Concise Introduction to Decentralized POMDPs

Learning Multiagent Communication with Backpropagation

Learning to Communicate with Deep Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning as a rehearsal for decentralized planning

Deep Exploration via Bootstrapped DQN

Multiagent cooperation and competition with deep reinforcement learning

Deep Reinforcement Learning with Double Q-Learning

Deep Recurrent Q-Learning for Partially Observable MDPs

Massively Parallel Methods for Deep Reinforcement Learning

Human-level control through deep reinforcement learning

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

The Arcade Learning Environment: An Evaluation Platform for General Agents

An Overview of Recent Progress in the Study of Distributed Multi-Agent Coordination

Incorporating Functional Knowledge in Neural Networks

Theoretical Advantages of Lenient Learners: An Evolutionary Game Theoretic Perspective

Optimal and Approximate Q-value Functions for Decentralized POMDPs

A Comprehensive Survey of Multiagent Reinforcement Learning

Hysteretic q-learning :an algorithm for decentralized reinforcement learning in cooperative multi-agent teams

Collaborative Multiagent Reinforcement Learning by Payoff Propagation

Multiagent Planning with Factored MDPs

An Algorithm for Distributed Reinforcement Learning in Cooperative Multi-Agent Systems

Approximation theory of the MLP model in neural networks

Long Short-Term Memory

Learning from delayed rewards

LIIR: Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning

Lenient Learning in Independent-Learner Stochastic Cooperative Games

Keepaway Soccer: From Machine Learning Testbed to Benchmark

Multiagent Reinforcement Learning for Multi-Robot Systems: A Survey