Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning (2021-02-06T00:00:00.000000Z)

TL;DR

The experiment results show that QMIX with normalized optimizations outperforms other works in SMAC, and beyond the common wisdom from these works, the monotonicity constraint can improve sample efficiency inSMAC and DEPP.

Abstract

Many complex multi-agent systems such as robot swarms control and autonomous vehicle coordination can be modeled as Multi-Agent Reinforcement Learning (MARL) tasks. QMIX, a widely popular MARL algorithm, has been used as a baseline for the benchmark environments, e.g., Starcraft Multi-Agent Challenge (SMAC), Difficulty-Enhanced Predator-Prey (DEPP). Recent variants of QMIX target relaxing the monotonicity constraint of QMIX, allowing for performance improvement in SMAC. In this paper, we investigate the code-level optimizations of these variants and the monotonicity constraint. (1) We find that such improvements of the variants are significantly affected by various code-level optimizations. (2) The experiment results show that QMIX with normalized optimizations outperforms other works in SMAC; (3) beyond the common wisdom from these works, the monotonicity constraint can improve sample efficiency in SMAC and DEPP. We also discuss why monotonicity constraints work well in purely cooperative tasks with a theoretical analysis. We open-source the code at \url{https://github.com/hijkzzz/pymarl2}.

Authors

Jian Hu

1 papers

Siyang Jiang

1 papers

Seth Austin Harding

1 papers

TL;DR

Abstract

Authors

References40 items

Revisiting Peng's Q(λ) for Modern Reinforcement Learning

SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving

Phasic Policy Gradient

QPLEX: Duplex Dueling Multi-Agent Q-Learning

Value-Decomposition Multi-Agent Actor-Critics

Off-Policy Multi-Agent Decomposed Policy Gradients

Learning Implicit Credit Assignment for Multi-Agent Actor-Critic

QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning

Weighted QMIX: Expanding Monotonic Value Function Factorisation

What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study

Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO

Macro-Action-Based Deep Multi-Agent Reinforcement Learning

FACMAC: Factored Multi-Agent Centralised Policy Gradients

Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning

MAVEN: Multi-Agent Variational Exploration

Deep Coordination Graphs

Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real

QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning

The StarCraft Multi-Agent Challenge

Multiagent Soft Q-Learning

QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Accelerated Methods for Deep Reinforcement Learning

Guided Deep Reinforcement Learning for Swarm Systems

Proximal Policy Optimization Algorithms

Value-Decomposition Networks For Cooperative Multi-Agent Learning

Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

Counterfactual Multi-Agent Policy Gradients

Multi-agent reinforcement learning as a rehearsal for decentralized planning

Asynchronous Methods for Deep Reinforcement Learning

Dueling Network Architectures for Deep Reinforcement Learning

Adam: A Method for Stochastic Optimization

Playing Atari with Deep Reinforcement Learning

An Overview of Recent Progress in the Study of Distributed Multi-Agent Coordination

Coordinated Multi-Agent Reinforcement Learning in Networked Distributed POMDPs

POMDPs for robotic tasks with mixed observability

Eligibility Traces for Off-Policy Policy Evaluation

Incremental multi-step Q-learning

Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson

Multi Agent Reinforcement Learning Independent vs Cooperative Agents

Reinforcement Learning: An Introduction

Field of Study