Git Re-Basin: Merging Models modulo Permutation Symmetries (2022-09-11T00:00:00.000000Z)

TL;DR

It is argued that neural network loss landscapes often contain (nearly) a single basin after accounting for all possible permutation symmetries of hidden units, and three algorithms to permute the units of one model to bring them into alignment with a reference model in order to merge the two models in weight space are introduced.

Abstract

The success of deep learning is due in large part to our ability to solve certain massive non-convex optimization problems with relative ease. Though non-convex optimization is NP-hard, simple algorithms -- often variants of stochastic gradient descent -- exhibit surprising effectiveness in fitting large neural networks in practice. We argue that neural network loss landscapes often contain (nearly) a single basin after accounting for all possible permutation symmetries of hidden units a la Entezari et al. 2021. We introduce three algorithms to permute the units of one model to bring them into alignment with a reference model in order to merge the two models in weight space. This transformation produces a functionally equivalent set of weights that lie in an approximately convex basin near the reference model. Experimentally, we demonstrate the single basin phenomenon across a variety of model architectures and datasets, including the first (to our knowledge) demonstration of zero-barrier linear mode connectivity between independently trained ResNet models on CIFAR-10. Additionally, we identify intriguing phenomena relating model width and training time to mode connectivity. Finally, we discuss shortcomings of the linear mode connectivity hypothesis, including a counterexample to the single basin theory.

Authors

S. Srinivasa

3 papers

Samuel K. Ainsworth

1 papers

J. Hayase

1 papers

TL;DR

Abstract

Authors

References103 items

REPAIR: REnormalizing Permuted Activations for Interpolation Repair

Random initialisations performing above chance and how to find them

Patching open-vocabulary models by interpolating weights

Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

On the Symmetries of Deep Learning Models and their Internal Representations

Linear Connectivity Reveals Generalization Strategies

Monotonic Differentiable Sorting Networks

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Deep networks on toroids: removing symmetries reveals the structure of flat regions in the landscape geometry

A ConvNet for the 2020s

Merging Models with Fisher-Weighted Averaging

Training Neural Networks with Fixed Sparse Masks

The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

Robust fine-tuning of zero-shot models

What can linear interpolation of neural network loss landscapes tell us?

Assessing Generalization of SGD via Disagreement

LLC: Accurate, Multi-purpose Learnt Low-dimensional Binary Codes

Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances

Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision

Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling

Learning Neural Network Subspaces

Faster Policy Learning with Continuous-Time Gradients

Revisiting "Qualitatively Characterizing Neural Network Optimization Problems"

Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth

Optimizing Mode Connectivity via Neuron Alignment

SoftSort: A Continuous Relaxation for the argsort Operator

Safe crossover of neural networks through neuron alignment

Fast Differentiable Sorting and Ranking

Federated Learning with Matched Averaging

Linear Mode Connectivity and the Lottery Ticket Hypothesis

Model Fusion via Optimal Transport

Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape

Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets

Differentiable Ranks and Sorting using Optimal Transport

Stochastic Optimization of Sorting Networks via Continuous Relaxations

Wide neural networks of any depth evolve as linear models under gradient descent

A Simple Baseline for Bayesian Uncertainty in Deep Learning

Are All Layers Created Equal?

oi-VAE: Output Interpretable VAEs for Nonlinear Group Factor Analysis

Neural Tangent Kernel: Convergence and Generalization in Neural Networks

On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport

A mean field view of the landscape of two-layer neural networks

Group Normalization

Averaging Weights Leads to Wider Optima and Better Generalization

Essentially No Barriers in Neural Network Energy Landscape

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

Learning Latent Permutations with Gumbel-Sinkhorn Networks

Visualizing the Loss Landscape of Neural Nets

An Introduction to Trajectory Optimization: How to Do Your Own Direct Collocation

Topology and Geometry of Half-Rectified Network Optimization

Federated Learning: Strategies for Improving Communication Efficiency

Federated Optimization: Distributed Machine Learning for On-Device Intelligence

SGDR: Stochastic Gradient Descent with Warm Restarts

On implementing 2D rectangular assignment algorithms

Instance Normalization: The Missing Ingredient for Fast Stylization

Layer Normalization

Deep Learning without Poor Local Minima

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

Communication-Efficient Learning of Deep Networks from Decentralized Data

BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

Top-N Recommender System via Matrix Completion

Deep Residual Learning for Image Recognition

Convergent Learning: Do different neural networks learn the same representations?

A method for finding similarity between multi-layer perceptrons by Forward Bipartite Alignment

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Qualitatively characterizing neural network optimization problems

How transferable are features in deep neural networks?

Very Deep Convolutional Networks for Large-Scale Image Recognition

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

Convex Relaxations for Permutation Problems

Maximum Quadratic Assignment Problem: Reduction from Maximum Label Cover and LP-based Approximation Algorithm

ImageNet: A large-scale hierarchical image database

Cryptosporidium Parvum Genome Project

On the Geometry of Feedforward Neural Network Error Surfaces

A shortest augmenting path algorithm for dense and sparse linear assignment problems

P-Complete Approximation Problems