Online Limited Memory Neural-Linear Bandits with Likelihood Matching (2021-02-07T00:00:00.000000Z)

TL;DR

This work has applied their algorithm, Limited Memory Neural-Linear with Likelihood Matching (NeuralLinear-LiM2) on a variety of datasets and observed that the algorithm achieves comparable performance to the unlimited memory approach while exhibits resilience to catastrophic forgetting.

Abstract

We study neural-linear bandits for solving problems where {\em both} exploration and representation learning play an important role. Neural-linear bandits harnesses the representation power of Deep Neural Networks (DNNs) and combines it with efficient exploration mechanisms by leveraging uncertainty estimation of the model, designed for linear contextual bandits on top of the last hidden layer. In order to mitigate the problem of representation change during the process, new uncertainty estimations are computed using stored data from an unlimited buffer. Nevertheless, when the amount of stored data is limited, a phenomenon called catastrophic forgetting emerges. To alleviate this, we propose a likelihood matching algorithm that is resilient to catastrophic forgetting and is completely online. We applied our algorithm, Limited Memory Neural-Linear with Likelihood Matching (NeuralLinear-LiM2) on a variety of datasets and observed that our algorithm achieves comparable performance to the unlimited memory approach while exhibits resilience to catastrophic forgetting.

Authors

Ofir Nabati

1 papers

Tom Zahavy

1 papers

Shie Mannor

3 papers

TL;DR

Abstract

Authors

References36 items

Neural Contextual Bandits with Deep Representation and Shallow Exploration

Every Model Learned by Gradient Descent Is Approximately a Kernel Machine

Neural Thompson Sampling

Bandit Algorithms

5分で分かる!? 有名論文ナナメ読み：Jacot, Arthor, Gabriel, Franck and Hongler, Clement : Neural Tangent Kernel : Convergence and Generalization in Neural Networks

Neural Contextual Bandits with UCB-based Exploration

VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning

Deep Randomized Least Squares Value Iteration

Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning

Randomized Prior Functions for Deep Reinforcement Learning

Efficient Exploration Through Bayesian Deep Q-Networks

The Uncertainty Bellman Equation and Exploration

A Tutorial on Thompson Sampling

Shallow Updates for Deep Reinforcement Learning

Curiosity-Driven Exploration by Self-Supervised Prediction

Overcoming catastrophic forgetting in neural networks

Is a picture worth a thousand words? A Deep Multi-Modal Fusion Architecture for Product Classification in e-commerce

Unifying Count-Based Exploration and Intrinsic Motivation

Mastering the game of Go with deep neural networks and tree search

A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification

Human-level control through deep reinforcement learning

Convolutional Neural Networks for Sentence Classification

Efficient Low-Rank Stochastic Gradient Descent Methods for Solving Semidefinite Programs

Thompson Sampling for Contextual Bandits with Linear Payoffs

Improved Algorithms for Linear Stochastic Bandits

Contextual Bandits with Linear Payoff Functions

A contextual-bandit approach to personalized news article recommendation

Distributed Inference for Latent Dirichlet Allocation

Using Confidence Bounds for Exploitation-Exploration Trade-offs

ON THE LIKELIHOOD THAT ONE UNKNOWN PROBABILITY EXCEEDS ANOTHER IN VIEW OF THE EVIDENCE OF TWO SAMPLES

Deep bayesian bandits showdown

and (6) An ablative version of (5) that calculates the prior only for the mean, similar to Levine et al.

Deep Learning

The Epoch-Greedy Algorithm for Multi-armed Bandits with Side Information

Semidefinite programming

Semide nite Programming

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names