A Closer Look at Invalid Action Masking in Policy Gradient Algorithms (2020-06-25T00:00:00.000000Z)

Abstract

In recent years, Deep Reinforcement Learning (DRL) algorithms have achieved state-of-the-art performance in many challenging strategy games. Because these games have complicated rules, an action sampled from the full discrete action distribution predicted by the learned policy is likely to be invalid according to the game rules (e.g., walking into a wall). The usual approach to deal with this problem in policy gradient algorithms is to “mask out” invalid actions and just sample from the set of valid actions. The implications of this process, however, remain under-investigated. In this paper, we 1) show theoretical justification for such a practice, 2) empirically demonstrate its importance as the space of invalid actions grows, and 3) provide further insights by evaluating different action masking regimes, such as removing masking after an agent has been trained using masking.

Authors

Shengyi Huang

2 papers

Santiago Ontan'on

1 papers

Abstract

Authors

References32 items

Implementation Matters in Deep RL: A Case Study on PPO and TRPO

Action Space Shaping in Deep Reinforcement Learning

Mastering Complex Control in MOBA Games with Deep Reinforcement Learning

Dota 2 with Large Scale Deep Reinforcement Learning

Grandmaster level in StarCraft II using multi-agent reinforcement learning

Comparing Observation and Action Representations for Deep Reinforcement Learning in MicroRTS

Learning Map-Independent Evaluation Functions for Real-Time Strategy Games

Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning

StarCraft II: A New Challenge for Reinforcement Learning

Proximal Policy Optimization Algorithms

אבן, and: Stone

Evaluating real-time strategy game states using convolutional neural networks

The Malmo Platform for Artificial Intelligence Experimentation

OpenAI Gym

Deep Reinforcement Learning in Large Discrete Action Spaces

Deep Reinforcement Learning in Parameterized Action Space

High-Dimensional Continuous Control Using Generalized Advantage Estimation

Adam: A Method for Stochastic Optimization

Rectified Linear Units Improve Restricted Boltzmann Machines

Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition

Policy Gradient Methods for Reinforcement Learning with Function Approximation

Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition

On Information and Sufficiency

On Information and Sufficiency

and Ba

Reinforcement Learning: An Introduction

Rewards Scaling 5 : Similarly, the reward is pre-processed by dividing the running variance of the discounted the returns, following by clipping it to a range

invalid action masking

Normalization of Advantages 3 : After calculating the advantages based on GAE, the advantages vector is normalized by subtracting its mean and divided by its standard deviation

For each update iteration in an epoch, the gradients of the policy and value network are clipped so that the

The Adam (Kingma and Ba 2015) optimizer's learning rate is set to decay as the number of timesteps agent trained increase

We now present the list of hyperparameters used in Table 4 and the list of neural network architecture in Table 5

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names