Discrete and Continuous Action Representation for Practical RL in Video Games (2019-12-23T00:00:00.000000Z)

TL;DR

It is shown that Hybrid SAC can successfully solve a highspeed driving task in one of the authors' games, and is competitive with the state-of-the-art on parameterized actions benchmark tasks.

Abstract

While most current research in Reinforcement Learning (RL) focuses on improving the performance of the algorithms in controlled environments, the use of RL under constraints like those met in the video game industry is rarely studied. Operating under such constraints, we propose Hybrid SAC, an extension of the Soft Actor-Critic algorithm able to handle discrete, continuous and parameterized actions in a principled way. We show that Hybrid SAC can successfully solve a highspeed driving task in one of our games, and is competitive with the state-of-the-art on parameterized actions benchmark tasks. We also explore the impact of using normalizing flows to enrich the expressiveness of the policy at minimal computational cost, and identify a potential undesired effect of SAC when used with normalizing flows, that may be addressed by optimizing a different objective.

Authors

Olivier Delalleau

2 papers

Maxim Peter

1 papers

Eloi Alonso

1 papers

TL;DR

Abstract

Authors

References31 items

Grandmaster level in StarCraft II using multi-agent reinforcement learning

Soft Actor-Critic for Discrete Action Settings

Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies

Leveraging exploration in off-policy algorithms via normalizing flows

Multi-Pass Q-Networks for Deep Reinforcement Learning with Parameterised Action Spaces

Distilling Policy Distillation

Discretizing Continuous Action Space for On-Policy Optimization

Soft Actor-Critic Algorithms and Applications

Relative Entropy Regularized Policy Iteration

Managing engineering systems with large state and action spaces through deep reinforcement learning

Hierarchical Approaches for Reinforcement Learning in Parameterized Action Space

Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space

Boosting Trust Region Policy Optimization by Normalizing Flows Policy

Latent Space Policies for Hierarchical Reinforcement Learning

Kickstarting Deep Reinforcement Learning

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Action Branching Architectures for Deep Reinforcement Learning

Rainbow: Combining Improvements in Deep Reinforcement Learning

Discrete Sequential Prediction of Continuous Actions for Deep RL

Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning

Deep Reinforcement Learning in Parameterized Action Space

Continuous control with deep reinforcement learning

Variational Inference with Normalizing Flows

Adam: A Method for Stochastic Optimization

The Arcade Learning Environment: An Evaluation Platform for General Agents

Using continuous action spaces to solve discrete problems

AlphaStar: Mastering the realtime strategy game StarCraft II

Discrete off-policy policy gradient using continuous relaxations

Pybullet, a python module for physics simulation for games, robotics and machine learning

Divergence measures based on the Shannon entropy

A set of D discrete components, with each component a di ( 1 ≤ i ≤ D ) being a discrete action taken in (1 , . . . , K i ). Such a representation has been used in particular by

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names