Attention in Psychology, Neuroscience, and Machine Learning (2020-04-16T00:00:00.000000Z)

TL;DR

Several use cases of attention in machine learning are covered, indicating their biological counterparts where they exist and the ways in which artificial attention can be further inspired by biology for the production of complex and integrative systems are explored.

Abstract

Attention is the important ability to flexibly control limited computational resources. It has been studied in conjunction with many other topics in neuroscience and psychology including awareness, vigilance, saliency, executive control, and learning. It has also recently been applied in several domains in machine learning. The relationship between the study of biological attention and its use as a tool to enhance artificial neural networks is not always clear. This review starts by providing an overview of how attention is conceptualized in the neuroscience and psychology literature. It then covers several use cases of attention in machine learning, indicating their biological counterparts where they exist. Finally, the ways in which artificial attention can be further inspired by biology for the production of complex and integrative systems is explored.

Authors

Grace W. Lindsay

2 papers

TL;DR

Abstract

Authors

References187 items

Multi-Task Learning By A Top-Down Control Network

The Costs and Benefits of Goal-Directed Attention in Deep Convolutional Neural Networks

Predicting Goal-directed Attention Control Using Inverse-Reinforcement Learning

Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future

A simple circuit model of visual cortex explains neural and behavioral aspects of attention

A unified circuit model of attention: Neural and behavioral effects

Cholinergic Modulation Promotes Attentional Modulation in Primary Visual Cortex- A Modeling Study

On the Relationship between Self-Attention and Convolutional Layers

Why Is There So Much More Research on Vision Than on Any Other Sensory Modality?

Introducing a Bayesian model of selective attention based on active inference

Subcortical connectivity correlates selectively with attention’s effects on spatial choice bias

No one knows what attention is

Saccader: Improving Accuracy of Hard Attention Models for Vision

Attention is not not Explanation

A flexible readout mechanism of human sensory representations

Conditional Computation for Continual Learning

Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)

Convolution, attention and structure embedding.

Attentive Single-Tasking of Multiple Tasks

An Attentive Survey of Attention Models

Holistic Reinforcement Learning: The Role of Structure and Attention

Many Task Learning With Task Routing

Attention is not Explanation

Combined loss of LAP1B and LAP1C results in an early onset multisystemic nuclear envelopathy

Attention, please! A Critical Review of Neural Attention Models in Natural Language Processing

Visual novelty, curiosity, and intrinsic reward in machine learning and the brain

A Modulation Module for Multi-task Learning with Applications in Image Retrieval

Learning what and where to attend

How biological attention mechanisms improve task performance in a large-scale visual system model

Top-Down Control of Visual Attention by the Prefrontal Cortex. Functional Specialization and Long-Range Interactions

The Consciousness Prior

Squeeze-and-Excitation Networks

Look Closer to See Better: Recurrent Attention Convolutional Neural Network for Fine-Grained Image Recognition

Modulating early visual processing by language

Attention is All you Need

Learning multiple visual domains with residual adapters

Using human brain activity to guide machine learning

SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning

Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

From Human Attention to Computational Attention

Weakly Supervised Part-of-speech Tagging Using Eye-tracking Data

End-to-End Instance Segmentation with Recurrent Attention

The Development of Attention Systems and Working Memory in Infancy

Long Short-Term Memory-Networks for Machine Reading

Pulvinar-Cortex Interactions in Vision and Attention

Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism

Look and Think Twice: Capturing Top-Down Visual Attention with Feedback Convolutional Neural Networks

Neuronal Mechanisms of Visual Attention.

ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering

A Source for Feature-Based Attention in the Prefrontal Cortex

Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering

Action Recognition using Visual Attention

Stacked Attention Networks for Image Question Answering

Effective Approaches to Attention-based Neural Machine Translation

Describing Multimedia Content Using Attention-Based Encoder-Decoder Networks

CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research

The negative priming paradigm: An update and implications for selective attention

The cocktail-party problem revisited: early processing and selection of multi-talker speech

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Selective attention affects conceptual object priming and recognition: a study with young and older adults

Attentional Neural Network: Feature Selection Using Cognitive Feedback

Neural Turing Machines

ImageNet Large Scale Visual Recognition Challenge

Neural Machine Translation by Jointly Learning to Align and Translate

Saliency-SVM: An automatic approach for image segmentation

Feature-based attentional modulation of orientation perception in somatosensation

Deep Networks with Internal Selective Attention through Feedback Connections

Early recurrent feedback facilitates visual object recognition under challenging conditions

Recurrent Models of Visual Attention

Psychostimulants and Cognition: A Continuum of Behavioral and Cognitive Activation

Information-seeking, curiosity, and attention: computational and neural mechanisms

Past rewards capture spatial attention and action choices

The Expected Value of Control: An Integrative Theory of Anterior Cingulate Cortex Function

Attention-dependent reductions in burstiness and action potential height in macaque area V4

Superior colliculus and visual spatial attention.

Attentional enhancement of spatial resolution: linking behavioural and neurophysiological evidence