NeurWIN: Neural Whittle Index Network For Restless Bandits Via Deep RL (2021-10-05T00:00:00.000000Z)

TL;DR

It is shown that a neural network that produces the Whittle index is also one that producing the optimal control for a set of Markov decision problems, which motivates using deep reinforcement learning for the training of NeurWIN.

Abstract

Whittle index policy is a powerful tool to obtain asymptotically optimal solutions for the notoriously intractable problem of restless bandits. However, finding the Whittle indices remains a difficult problem for many practical restless bandits with convoluted transition kernels. This paper proposes NeurWIN, a neural Whittle index network that seeks to learn the Whittle indices for any restless bandits by leveraging mathematical properties of the Whittle indices. We show that a neural network that produces the Whittle index is also one that produces the optimal control for a set of Markov decision problems. This property motivates using deep reinforcement learning for the training of NeurWIN. We demonstrate the utility of NeurWIN by evaluating its performance for three recently studied restless bandit problems. Our experiment results show that the performance of NeurWIN is significantly better than other RL algorithms.

Authors

Khaled Nakhleh

1 papers

Santosh Ganji

2 papers

Ping-Chun Hsieh

1 papers

TL;DR

Abstract

Authors

References38 items

Learn to Intervene: An Adaptive Learning Policy for Restless Bandits in Application to Preventive Healthcare

A single algorithm for both restless and rested rotting bandits

MOPO: Model-based Offline Policy Optimization

MOReL : Model-Based Offline Reinforcement Learning

Whittle index based Q-learning for restless bandits with average reward

Q-Learning in enormous action spaces via amortized approximate maximization

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Multi-Armed Bandits: Theory and Applications to Online Learning in Networks

Towards Q-learning the Whittle Index for Restless Bandits

Recovering Bandits

A learning algorithm for the Whittle index policy for scheduling web crawlers

A Whittle Index Approach to Minimizing Functions of Age of Information

Deep Reinforcement Learning for Wireless Sensor Scheduling in Cyber-Physical Systems

Problem Dependent Reinforcement Learning Bounds Which Can Identify Bandit Structure in MDPs

Deep Reinforcement Learning for Dynamic Multichannel Access in Wireless Networks

Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling

Deep reinforcement learning for building HVAC control

Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learning

Contextual Decision Processes with low Bellman rank are PAC-Learnable

Deadline scheduling as restless bandits

On the Whittle Index for Restless Multiarmed Hidden Markov Bandits

Deep Reinforcement Learning in Large Discrete Action Spaces

When are Kalman-Filter Restless Bandits Indexable?

Whittle Index Approach to Size-aware Scheduling with Time-varying Channels

Adam: A Method for Stochastic Optimization

Convergence Results for Some Temporal Difference Methods Based on Least Squares

Adapting to a Changing Environment: the Brownian Restless Bandits

Multi-UAV dynamic routing with partial observations using restless bandit allocation indices

Dynamic priority allocation via restless bandit marginal productivity indices

Learning Algorithms for Markov Decision Processes with Average Cost

The Complexity of Optimal Queuing Network Control

ChangyWen

Learning reinforcement learning: Reinforce with pytorch! https://towardsdatascience.com/learning-reinforcement-learning-reinforce-with-pytorch- 5e8ad7fc7da0, Dec 2019

A reinforcement learning algorithm for restless bandits

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Reinforcement Learning: An Introduction

Restless bandits: activity allocation in a changing world

3000 timesteps, discount factor β = 0 . 999 ,

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names