Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
Published in
arXiv.org(2020)External Links:
TL
TL;DR
It is demonstrated that Independent PPO (IPPO), a form of independent learning in which each agent simply estimates its local value function, can perform just as well as or better than state-of-the-art joint learning approaches on popular multi-agent benchmark suite SMAC with little hyperparameter tuning.
Abstract
Most recently developed approaches to cooperative multi-agent reinforcement learning in the \emph{centralized training with decentralized execution} setting involve estimating a centralized, joint value function. In this paper, we demonstrate that, despite its various theoretical shortcomings, Independent PPO (IPPO), a form of independent learning in which each agent simply estimates its local value function, can perform just as well as or better than state-of-the-art joint learning approaches on popular multi-agent benchmark suite SMAC with little hyperparameter tuning. We also compare IPPO to several variants; the results suggest that IPPO's strong performance may be due to its robustness to some forms of environment non-stationarity.
References35 items
1
Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms on a Building Energy Demand Coordination Task
2
Revisiting Design Choices in Proximal Policy Optimization
3
Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
4
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
5