MAVEN: Multi-Agent Variational Exploration (2019-10-16T00:00:00.000000Z)

TL;DR

This paper analyses value-based methods that are known to have superior performance in complex environments and proposes a novel approach called MAVEN that hybridises value and policy-based methods by introducing a latent space for hierarchical control.

Abstract

Centralised training with decentralised execution is an important setting for cooperative deep multi-agent reinforcement learning due to communication constraints during execution and computational tractability in training. In this paper, we analyse value-based methods that are known to have superior performance in complex environments [43]. We specifically focus on QMIX [40], the current state-of-the-art in this domain. We show that the representational constraints on the joint action-values introduced by QMIX and similar methods lead to provably poor exploration and suboptimality. Furthermore, we propose a novel approach called MAVEN that hybridises value and policy-based methods by introducing a latent space for hierarchical control. The value-based agents condition their behaviour on the shared latent variable controlled by a hierarchical policy. This allows MAVEN to achieve committed, temporally extended exploration, which is key to solving complex multi-agent tasks. Our experimental results show that MAVEN achieves significant performance improvements on the challenging SMAC domain [43].

Authors

Tabish Rashid

5 papers

Mikayel Samvelyan

7 papers

Shimon Whiteson

13 papers

TL;DR

Abstract

Authors

References53 items

QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning

The StarCraft Multi-Agent Challenge

CESMA: Centralized Expert Supervises Multi-Agents

VIREL: A Variational Inference Framework for Reinforcement Learning

New And Surprising Ways to Be Mean

Learning Attentional Communication for Multi-Agent Cooperation

Latent Space Policies for Hierarchical Reinforcement Learning

QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Diversity is All You Need: Learning Skills without a Reward Function

Structured Exploration via Hierarchical Variational Policy Networks

Coordinated Exploration in Concurrent Reinforcement Learning

Guided Deep Reinforcement Learning for Swarm Systems

Value-Decomposition Networks For Cooperative Multi-Agent Learning

Symmetry Learning for Function Approximation in Reinforcement Learning

Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

Counterfactual Multi-Agent Policy Gradients

Curiosity-Driven Exploration by Self-Supervised Prediction

Symmetry Detection and Exploitation for Function Approximation in Deep RL

Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games

Deep Exploration via Randomized Value Functions

Deep Decentralized Multi-task Multi-Agent RL under Partial Observability

Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning

Variational Intrinsic Control

Stochastic Neural Networks for Hierarchical Reinforcement Learning

Improved Variational Inference with Inverse Autoregressive Flow

A Concise Introduction to Decentralized POMDPs

Learning Multiagent Communication with Backpropagation

Learning to Communicate with Deep Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning as a rehearsal for decentralized planning

VIME: Variational Information Maximizing Exploration

Deep Exploration via Bootstrapped DQN

Multiagent cooperation and competition with deep reinforcement learning

Deep Recurrent Q-Learning for Partially Observable MDPs

Variational Inference with Normalizing Flows

Neural Variational Inference and Learning in Belief Networks

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Stochastic Gradient VB and the Variational Auto-Encoder

Playing Atari with Deep Reinforcement Learning

An Overview of Recent Progress in the Study of Distributed Multi-Agent Coordination

Bayesian Time Series Models

Convex optimization

Graphical Models, Exponential Families, and Variational Inference

Maximum Entropy Inverse Reinforcement Learning

A Comprehensive Survey of Multiagent Reinforcement Learning

Linearly-solvable Markov decision problems

Pattern Recognition and Machine Learning

Efficient Solution Algorithms for Factored MDPs

Multiagent Planning with Factored MDPs

Long Short-Term Memory

Subjectivity and Correlation in Randomized Strategies

Visualizing Data using t-SNE

Multiagent Reinforcement Learning for Multi-Robot Systems: A Survey

Multi Agent Reinforcement Learning Independent vs Cooperative Agents

Field of Study

Venue Information

Name

Type

URL

Alternate Names