The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks (2021-10-12T00:00:00.000000Z)

TL;DR

If the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them, which has implications for lottery ticket hypothesis, distributed training, and ensemble methods.

Abstract

In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them. Although it is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We further provide a preliminary theoretical result to support our conjecture. Our conjecture has implications for lottery ticket hypothesis, distributed training, and ensemble methods.

Authors

Behnam Neyshabur

4 papers

R. Entezari

1 papers

Hanie Sedghi

1 papers

TL;DR

Abstract

Authors

References45 items

Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances

Learning Neural Network Subspaces

Caliban: Docker-based job manager for reproducible workflows

Optimizing Mode Connectivity via Neuron Alignment

What is being transferred in transfer learning?

Towards Learning Convolutions from Scratch

Loss landscapes and optimization in over-parameterized non-linear systems and neural networks

BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning

Linear Mode Connectivity and the Lottery Ticket Hypothesis

Deep double descent: where bigger models and more data hurt

Implicit Regularization and Convergence for Weight Normalization

Model Fusion via Optimal Transport

Deep Ensembles: A Loss Landscape Perspective

Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape

Large Scale Structure of Neural Network Loss Landscapes

Shaping the learning landscape in neural networks around wide flat minima

Uniform convergence may be unable to explain generalization in deep learning

Reconciling modern machine-learning practice and the classical bias–variance trade-off

Over-Parameterized Deep Neural Networks Have No Strict Local Minima For Any Continuous Activations

On the loss landscape of a class of deep neural networks with no bad local valleys

The jamming transition as a paradigm to understand the loss landscape of deep neural networks

Neural Tangent Kernel: Convergence and Generalization in Neural Networks

Multi-Task Zipping via Layer-wise Neuron Sharing

A mean field view of the landscape of two-layer neural networks

Averaging Weights Leads to Wider Optima and Better Generalization

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Essentially No Barriers in Neural Network Energy Landscape

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

Visualizing the Loss Landscape of Neural Nets

Exploring Generalization in Deep Learning

Topology and Geometry of Half-Rectified Network Optimization

Understanding deep learning requires rethinking generalization

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

List-Based Simulated Annealing Algorithm for Traveling Salesman Problem

Deep Residual Learning for Image Recognition

Path-SGD: Path-Normalized Optimization in Deep Neural Networks

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet: A large-scale hierarchical image database

Local minima and plateaus in hierarchical structures of multilayer perceptrons

PRINCIPLES OF NEURODYNAMICS. PERCEPTRONS AND THE THEORY OF BRAIN MECHANISMS

Optimal Convergence Rates for Convex Distributed Optimization in Networks

Reading Digits in Natural Images with Unsupervised Feature Learning

MNIST handwritten digit database. http://yann.lecun.com/exdb/mnist

Cifar-100 and cifar-10

Principles of neurodynamics

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names