Signal Novelty Detection as an Intrinsic Reward for Robotics (2023-04-01T00:00:00.000000Z)

TL;DR

An Autoencoder deep learning neural network was utilized as novelty detection for intrinsic rewards to guide the search process through a state space and achieved more efficient and accurate robot control in three of the four tasks when purely intrinsic rewards were used compared to standard extrinsic rewards.

Abstract

In advanced robot control, reinforcement learning is a common technique used to transform sensor data into signals for actuators, based on feedback from the robot’s environment. However, the feedback or reward is typically sparse, as it is provided mainly after the task’s completion or failure, leading to slow convergence. Additional intrinsic rewards based on the state visitation frequency can provide more feedback. In this study, an Autoencoder deep learning neural network was utilized as novelty detection for intrinsic rewards to guide the search process through a state space. The neural network processed signals from various types of sensors simultaneously. It was tested on simulated robotic agents in a benchmark set of classic control OpenAI Gym test environments (including Mountain Car, Acrobot, CartPole, and LunarLander), achieving more efficient and accurate robot control in three of the four tasks (with only slight degradation in the Lunar Lander task) when purely intrinsic rewards were used compared to standard extrinsic rewards. By incorporating autoencoder-based intrinsic rewards, robots could potentially become more dependable in autonomous operations like space or underwater exploration or during natural disaster response. This is because the system could better adapt to changing environments or unexpected situations.

Authors

Jiri Pospíchal

2 papers

Martin Kubovčík

2 papers

Iveta Dirgová Luptáková

2 papers

TL;DR

Abstract

Authors

References61 items

Intrinsic Rewards from Self-Organizing Feature Maps for Exploration in Reinforcement Learning

Occupancy Reward-Driven Exploration with Deep Reinforcement Learning for Mobile Robot System

Auto-Encoder-Enabled Anomaly Detection in Acceleration Data: Use Case Study in Container Handling Operations

GAN-based Intrinsic Exploration for Sample Efficient Reinforcement Learning

Deep reinforcement learning-based safe interaction for industrial human-robot collaboration using intrinsic reward function

Evaluating the Use of Reconstruction Error for Novelty Localization

Application of the Deep CNN-Based Method in Industrial System for Wire Marking Identification

Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model*

Toward Deep Supervised Anomaly Detection: Reinforcement Learning from Partially Labeled Anomaly Data

Revisiting Fundamentals of Experience Replay

AESMOTE: Adversarial Reinforcement Learning With SMOTE for Anomaly Detection

A sample efficient model-based deep reinforcement learning algorithm with experience replay for robot manipulation

End-to-End Navigation Strategy With Deep Reinforcement Learning for Mobile Robots

Agent57: Outperforming the Atari Human Benchmark

Never Give Up: Learning Directed Exploration Strategies

End-to-end Deep Reinforcement Learning for Multi-agent Collaborative Exploration

Q-Learning Algorithms: A Comprehensive Classification and Applications

Dynamic Stale Synchronous Parallel Distributed Training for Deep Learning

Learning-Driven Exploration for Reinforcement Learning

Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning

強化学習におけるVariational Information Maximizing Explorationに基づく状態探索の効率化

Learning Representations in Model-Free Hierarchical Reinforcement Learning

Exploration by Random Network Distillation

DORA The Explorer: Directed Outreaching Reinforcement Action-Selection

Time Limits in Reinforcement Learning

Rainbow: Combining Improvements in Deep Reinforcement Learning

Curiosity-Driven Exploration by Self-Supervised Prediction

#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning

Unifying Count-Based Exploration and Intrinsic Motivation

OpenAI Gym

VIME: Variational Information Maximizing Exploration

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Dueling Network Architectures for Deep Reinforcement Learning

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

Playing Atari with Deep Reinforcement Learning

Analysis of performance bottleneck of P2P grid applications

Global Path Planning on Board the Mars Exploration Rovers

Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding

Neuronlike adaptive elements that can solve difficult learning control problems

State-of-the-Art; Springer: Berlin/Heidelberg, Germany, 2012; pp

Tune Hyperparameters

Cambridge, UK, 1990

16th International Work-Conference on Artificial Neural Networks, IWANN 2021, Virtual Event

Theory and Numerical Examples

Optimization of the Functional Decomposition of Parallel and Distributed Computations in Graph Coloring With the Use of High-Performance Computing

Research on the development and path exploration of autonomous underwater robots

CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms

Heat-RL: Online Model Selection for Streaming Time-Series Anomaly Detection

Stable-Baselines3: Reliable Reinforcement Learning Implementations

Autonomous Docking of Mobile Robots by Reinforcement Learning Tackling the Sparse Reward Problem

Gymnasium

Reinforcement Learning and Markov Decision Processes

Convolutional Deep Belief Networks on CIFAR-10

Visualizing Data using t-SNE

Gradient-based learning applied to document recognition

Reinforcement Learning: An Introduction

Efficient memory-based learning for robot control

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION 1 Intrinsic Motivation Systems for Autonomous Mental Development

Reinforcement Learning in Non-Stationary Inﬁnite Horizon Environments/submitted by Christian Alexander Steinparz. BSc. Master’s Thesis, Johannes Kepler Universität Linz, Linz, Austria, 2021

DQN Agent Playing LunarLander-v2

Dying Relu and Initialization : Theory and Numerical Examples . 2019

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names