End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training (2024-02-14T00:00:00.000000Z)

TL;DR

The results of the normalized HSIC value analysis reveal the E2E training ability to exhibit different information dynamics across layers, in addition to efficient information propagation, and suggest the need to consider the cooperative interactions between layers, not just the final layer when analyzing the information bottleneck of deep learning.

Abstract

End-to-end (E2E) training, optimizing the entire model through error backpropagation, fundamentally supports the advancements of deep learning. Despite its high performance, E2E training faces the problems of memory consumption, parallel computing, and discrepancy with the functionalities of the actual brain. Various alternative methods have been proposed to overcome these difficulties; however, no one can yet match the performance of E2E training, thereby falling short in practicality. Furthermore, there is no deep understanding regarding differences in the trained model properties beyond the performance gap. In this paper, we reconsider why E2E training demonstrates a superior performance through a comparison with layer-wise training, a non-E2E method that locally sets errors. On the basis of the observation that E2E training has an advantage in propagating input information, we analyze the information plane dynamics of intermediate representations based on the Hilbert-Schmidt independence criterion (HSIC). The results of our normalized HSIC value analysis reveal the E2E training ability to exhibit different information dynamics across layers, in addition to efficient information propagation. Furthermore, we show that this layer-role differentiation leads to the final representation following the information bottleneck principle. It suggests the need to consider the cooperative interactions between layers, not just the final layer when analyzing the information bottleneck of deep learning.

Authors

Keitaro Sakamoto

1 papers

Issei Sato

1 papers

TL;DR

Abstract

Authors

References97 items

Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination

Automatic Network Pruning via Hilbert-Schmidt Independence Criterion Lasso under Information Bottleneck Principle

The Tunnel Effect: Building Data Representations in Deep Neural Networks

How Does Information Bottleneck Help Deep Learning?

DualHSIC: HSIC-Bottleneck and Alignment for Continual Learning

Information Plane Analysis for Dropout Neural Networks

Blockwise Self-Supervised Learning at Scale

The Forward-Forward Algorithm: Some Preliminary Investigations

Pruning Adversarially Robust Neural Networks without Adversarial Examples

Hebbian Deep Learning Without Feedback

Improving the Trainability of Deep Neural Networks through Layerwise Batch-Entropy Regularization

Signal Propagation: The Framework for Learning and Inference in a Forward Pass

Gradients without Backpropagation

Evolving Decomposed Plasticity Rules for Information-Bottlenecked Meta-Learning

SoftHebb: Bayesian inference in unsupervised Hebbian soft winner-take-all networks

Information Bottleneck: Exact Analysis of (Quantized) Neural Networks

Self-Supervised Learning with Kernel Dependence Maximization

Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustness

Revisiting Locally Supervised Learning: an Alternative to End-to-end Training

Pruning and Quantization for Deep Neural Network Acceleration: A Survey

Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methods

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Interlocking Backpropagation: Improving depthwise model-parallelism

LoCo: Local Contrastive Representation Learning

Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks

Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere

Supervised Contrastive Learning

On Information Plane Analyses of Neural Network Classifiers—A Review

What Information Does a ResNet Compress?

What is the State of Neural Network Pruning?

Learning Robust Representations via Multi-View Information Bottleneck

A Simple Framework for Contrastive Learning of Visual Representations

The HSIC Bottleneck: Deep Learning without Back-Propagation

On Mutual Information Maximization for Representation Learning

Feature Selection via Mutual Information: New Theoretical Insights

Associated Learning: Decomposing End-to-End Backpropagation Based on Autoencoders and Target Propagation

Contrastive Multiview Coding

Putting An End to End-to-End: Gradient-Isolated Learning of Representations

A Graph Theoretic Framework of Recomputation Algorithms for Memory-Efficient Backpropagation

Adaptive Estimators Show Information Compression in Deep Neural Networks

Are All Layers Created Equal?

Analyzing and Improving Representations with the Soft Nearest Neighbor Loss

Decoupled Greedy Learning of CNNs

Training Neural Networks with Local Error Signals

Greedy Layerwise Learning Can Scale to ImageNet

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

Estimating Information Flow in Deep Neural Networks

Learning deep representations by mutual information estimation and maximization

Quantizing deep convolutional networks for efficient inference: A whitepaper

Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures

Deep Cascade Learning

Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle

On the information bottleneck theory of deep learning

Deep Supervised Learning Using Local Errors

Learning in the machine: The symmetries of the deep learning channel

Learning Deep ResNet Blocks Sequentially using Boosting Theory

Attention is All you Need

Emergence of Invariance and Disentanglement in Deep Representations

Layer-wise training of deep networks using kernel similarity

Universal Adversarial Perturbations

Understanding intermediate layers using linear classifier probes

Deep Incremental Boosting

Direct Feedback Alignment Provides Learning in Deep Neural Networks

Residual Networks Behave Like Ensembles of Relatively Shallow Networks

Training Deep Nets with Sublinear Memory Cost

Deep Residual Learning for Image Recognition

Deep learning and the information bottleneck principle

Difference Target Propagation

Explaining and Harnessing Adversarial Examples

Random feedback weights support learning in deep neural networks

Very Deep Convolutional Networks for Large-Scale Image Recognition

How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation

One weird trick for parallelizing convolutional neural networks

The Basics of Brain Development

Universality, Characteristic Kernels and RKHS Embedding of Measures

Kernel dimension reduction in regression