QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020-08-03T00:00:00.000000Z)

TL;DR

A novel MARL approach, called duPLEX dueling multi-agent Q-learning (QPLEX), which takes a duplex dueling network architecture to factorize the joint value function and encodes the IGM principle into the neural network architecture and thus enables efficient value function learning.

Abstract

We explore value-based multi-agent reinforcement learning (MARL) in the popular paradigm of centralized training with decentralized execution (CTDE). CTDE has an important concept, Individual-Global-Max (IGM) principle, which requires the consistency between joint and local action selections to support efficient local decision-making. However, in order to achieve scalability, existing MARL methods either limit representation expressiveness of their value function classes or relax the IGM consistency, which may suffer from instability risk or lead to poor performance. This paper presents a novel MARL approach, called duPLEX dueling multi-agent Q-learning (QPLEX), which takes a duplex dueling network architecture to factorize the joint value function. This duplex dueling structure encodes the IGM principle into the neural network architecture and thus enables efficient value function learning. Theoretical analysis shows that QPLEX achieves a complete IGM function class. Empirical experiments on StarCraft II micromanagement tasks demonstrate that QPLEX significantly outperforms state-of-the-art baselines in both online and offline data collection settings, and also reveal that QPLEX achieves high sample efficiency and can benefit from offline datasets without additional online exploration.

Authors

Yang Yu

2 papers

Jianhao Wang

1 papers

Zhizhou Ren

1 papers

TL;DR

Abstract

Authors

References36 items

RODE: Learning Roles to Decompose Multi-Agent Tasks

Off-Policy Multi-Agent Decomposed Policy Gradients

An Optimistic Perspective on Offline Deep Reinforcement Learning

Weighted QMIX: Expanding Monotonic Value Function Factorisation

Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning

MOPO: Model-based Offline Policy Optimization

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

D4RL: Datasets for Deep Data-Driven Reinforcement Learning

ROMA: Multi-Agent Reinforcement Learning with Emergent Roles

Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning

MAVEN: Multi-Agent Variational Exploration

Influence-Based Multi-Agent Exploration

Learning Nearly Decomposable Value Functions Via Communication Minimization

Benchmarking Batch Deep Reinforcement Learning Algorithms

An Optimistic Perspective on Offline Reinforcement Learning

Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction

QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning

Challenges of Real-World Reinforcement Learning

The StarCraft Multi-Agent Challenge

Off-Policy Deep Reinforcement Learning without Exploration

Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward

QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Guided Deep Reinforcement Learning for Swarm Systems

Attention is All you Need

A Concise Introduction to Decentralized POMDPs

Multi-agent reinforcement learning as a rehearsal for decentralized planning

Dueling Network Architectures for Deep Reinforcement Learning

MuJoCo: A physics engine for model-based control

The Arcade Learning Environment: An Evaluation Platform for General Agents

An Overview of Recent Progress in the Study of Distributed Multi-Agent Coordination

Coordinated Multi-Agent Reinforcement Learning in Networked Distributed POMDPs

Optimal and Approximate Q-value Functions for Decentralized POMDPs

Cryptosporidium Parvum Genome Project

Planning, Learning and Coordination in Multiagent Decision Processes

Multi Agent Reinforcement Learning Independent vs Cooperative Agents

Approximation with artificial neural networks

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names