Hopfield Networks is All You Need (2020-07-16T00:00:00.000000Z)

TL;DR

A new PyTorch layer is provided, called "Hopfield", which allows to equip deep learning architectures with modern Hopfield networks as a new powerful concept comprising pooling, memory, and attention.

Abstract

We show that the transformer attention mechanism is the update rule of a modern Hopfield network with continuous states. This new Hopfield network can store exponentially (with the dimension) many patterns, converges with one update, and has exponentially small retrieval errors. The number of stored patterns is traded off against convergence speed and retrieval error. The new Hopfield network has three types of energy minima (fixed points of the update): (1) global fixed point averaging over all patterns, (2) metastable states averaging over a subset of patterns, and (3) fixed points which store a single pattern. Transformer and BERT models operate in their first layers preferably in the global averaging regime, while they operate in higher layers in metastable states. The gradient in transformers is maximal for metastable states, is uniformly distributed for global averaging, and vanishes for a fixed point near a stored pattern. Using the Hopfield network interpretation, we analyzed learning of transformer and BERT models. Learning starts with attention heads that average and then most of them switch to metastable states. However, the majority of heads in the first layers still averages and can be replaced by averaging, e.g. our proposed Gaussian weighting. In contrast, heads in the last layers steadily learn and seem to use metastable states to collect information created in lower layers. These heads seem to be a promising target for improving transformers. Neural networks with Hopfield networks outperform other methods on immune repertoire classification, where the Hopfield net stores several hundreds of thousands of patterns. We provide a new PyTorch layer called "Hopfield", which allows to equip deep learning architectures with modern Hopfield networks as a new powerful concept comprising pooling, memory, and attention. GitHub: this https URL

References107 items

Could graph neural networks learn better molecular representation for drug discovery? A comparison study of descriptor-based and graph-based models

Pushing the boundaries of molecular representation for drug discovery with graph attention mechanism.

Large Associative Memory Problem in Neurobiology and Machine Learning

Set Distribution Networks: a Generative Model for Sets of Images

Synthesizer: Rethinking Self-Attention for Transformer Models

Modern Hopfield Networks and Attention for Immune Repertoire Classification

Encoding-based Memory Modules for Recurrent Neural Networks

MEMO: A Deep Network for Flexible Combination of Episodic Memories

PyTorch: An Imperative Style, High-Performance Deep Learning Library

HuggingFace's Transformers: State-of-the-art Natural Language Processing

Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving

A compact vocabulary of paratope-epitope interactions enables predictability of antibody-antigen binding

immuneSIM: tunable multi-feature simulation of B- and T-cell receptor repertoires for immunoinformatics benchmarking

Release Strategies and the Social Impacts of Language Models

Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)

Learning to Reason with Third-Order Tensor Products

Sharp bounds for the Lambert W function

A new mechanical approach to handle generalized Hopfield neural networks

Bag encoding strategies in multiple instance learning problems

Study and Observation of the Variation of Accuracies of KNN, SVM, LMNN, ENN Algorithms on Eleven Different Datasets from UCI Machine Learning Repository

Universal Transformers

SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters

BRUNO: A Deep Recurrent Model for Exchangeable Data

Attention-based Deep Multiple Instance Learning

Analysis on Polish Spaces and an Introduction to Optimal Transportation

Decoupled Weight Decay Regularization

Automatic differentiation in PyTorch

Learning to update Auto-associative Memory in Recurrent Neural Networks for Improving Sequence Memorization

Attention is All you Need

Self-Normalizing Neural Networks

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Neural Message Passing for Quantum Chemistry

On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning

Immunosequencing identifies signatures of cytomegalovirus exposure history and HLA-mediated effects on the T cell repertoire

Deep Sets

Frustratingly Short Attention Spans in Neural Language Modeling

On a Model of Associative Memory with Huge Storage Capacity

On the Maximum Storage Capacity of the Hopfield Model

Dense Associative Memory Is Robust to Adversarial Inputs

Permutation-equivariant neural networks applied to dynamics prediction

Multiple instance learning: A survey of problem characteristics and applications

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Using Fast Weights to Attend to the Recent Past

Computational Modeling of β-Secretase 1 (BACE-1) Inhibitors Using Ligand Based Approaches

Revisiting multiple instance neural networks

Pointer Sentinel Mixture Models

Semi-Supervised Classification with Graph Convolutional Networks

The complexity model of communication with computer images

Dense Associative Memory for Pattern Recognition

Variations and extension of the convex–concave procedure

XGBoost: A Scalable Tree Boosting System

Associative Long Short-Term Memory

Random Point Sets on the Sphere—Hole Radii, Covering, and Separation

The SIDER database of drugs and side effects

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

End-To-End Memory Networks

Neural Turing Machines

Memory Networks

Empowering Multiple Instance Histopathology Cancer Diagnosis by Cell Graphs

Neural Machine Translation by Jointly Learning to Align and Translate

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Deep learning in neural networks: An overview

Dissimilarity-Based Ensembles for Multiple Instance Learning

Distributions of angles in random packing on spheres

Topological and dynamical complexity of random neural networks.

A Bayesian Approach to in Silico Blood-Brain Barrier Penetration Modeling

Convex Analysis and Monotone Operator Theory in Hilbert Spaces

NIST Handbook of Mathematical Functions

Convex optimization

On the Convergence of the Concave-Convex Procedure

Et al

MILES: Multiple-Instance Learning via Embedded Instance Selection

An isotropic Gaussian mixture can have more modes than components

The Concave-Convex Procedure

Storage capacity of attractor neural networks with depressing synapses.

Learning with Kernels: support vector machines, regularization, optimization, and beyond

The Concave-Convex Procedure (CCCP)

Solving the Multiple-Instance Problem: A Lazy Learning Approach

Convergence Properties of the Softassign Quadratic Assignment Algorithm

A Framework for Multiple-Instance Learning

Long Short-Term Memory

On the Storage Capacity of Nonlinear Neural Networks

Solving the Multiple Instance Problem with Axis-Parallel Rectangles

A Novel Optimizing Network Architecture with Applications

Introduction to the theory of neural computation

Dynamics of Discrete Time, Continuous State Hopfield Networks

Mining association rules between sets of items in large databases

On the number of spurious memories in the Hopfield model

The capacity of the Hopfield associative memory

Saturation Level of the Hopfield Model for Neural Network

Neurons with graded response have collective computational properties like those of two-state neurons.

ON THE CONVERGENCE PROPERTIES OF THE EM ALGORITHM

Neural networks and physical systems with emergent collective computational abilities.

Analytic theory of the ground state properties of a spin glass. II. XY spin glass

Sufficient Conditions for the Convergence of Monotonic Mathematical Programming Algorithms

Nonlinear Programming: A Unified Approach.

Electra: Pre-training text encoders as discriminators rather than generators

Deep multiple instance learning for digital histopathology

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

100

Improved Expressivity Through Dendritic Neural Networks

101

Are Random Forests Truly the Best Classifiers?

102

Do we need hundreds of classifiers to solve real world classification problems?

103

INEQUALITIES ON THE LAMBERTW FUNCTION AND HYPERPOWER FUNCTION

104

Support Vector Machines for Multiple-Instance Learning

105

Information Capacity o f the Hopfie ld Mode l

106

Reinforcement Learning: An Introduction

107

Untersuchungen zu dynamischen neuronalen Netzen