Layerwise Linear Mode Connectivity (2023-07-13T00:00:00.000000Z)

TL;DR

A novel notion of the layer-wise linear connectivity is introduced, and it is shown that deep networks do not have layer-wise barriers between them.

Abstract

Averaging neural network parameters is an intuitive method for fusing the knowledge of two independent models. It is most prominently used in federated learning. If models are averaged at the end of training, this can only lead to a good performing model if the loss surface of interest is very particular, i.e., the loss in the midpoint between the two models needs to be sufficiently low. This is impossible to guarantee for the non-convex losses of state-of-the-art networks. For averaging models trained on vastly different datasets, it was proposed to average only the parameters of particular layers or combinations of layers, resulting in better performing models. To get a better understanding of the effect of layer-wise averaging, we analyse the performance of the models that result from averaging single layers, or groups of layers. Based on our empirical and theoretical investigation, we introduce a novel notion of the layer-wise linear connectivity, and show that deep networks do not have layer-wise barriers between them.

Authors

Maksym Andriushchenko

2 papers

Linara Adilova

1 papers

Michael Kamp

1 papers

TL;DR

Abstract

Authors

References58 items

Can Neural Network Memorization Be Localized?

Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity

Normalization Layers Are All That Sharpness-Aware Minimization Needs

Stochastic collapse: how gradient noise attracts SGD dynamics towards simpler subnetworks

Decentralized SGD and Average-direction SAM are Asymptotically Equivalent

Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoff

Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

A modern look at the relationship between sharpness and generalization

CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNet

Editing Models with Task Arithmetic

Mechanistic Mode Connectivity

REPAIR: REnormalizing Permuted Activations for Interpolation Repair

How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization

SGD with large step sizes learns sparse features

Revisiting adapters with adversarial training

Git Re-Basin: Merging Models modulo Permutation Symmetries

On the Importance and Applicability of Pre-Training for Federated Learning

Layer-wised Model Aggregation for Personalized Federated Learning

Federated Learning with Partial Model Personalization

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

What can linear interpolation of neural network loss landscapes tell us?

Revisiting Model Stitching to Compare Neural Representations

FedBABU: Towards Enhanced Representation for Federated Image Classification

Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances

Learning Neural Network Subspaces

FedBN: Federated Learning on Non-IID Features via Local Batch Normalization

Sharpness-Aware Minimization for Efficiently Improving Generalization

The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks

Federated Learning with Matched Averaging

Notes on the Symmetries of 2-Layer ReLU-Networks

Think Locally, Act Globally: Federated Learning with Local and Global Representations

Relative Flatness and Generalization

Linear Mode Connectivity and the Lottery Ticket Hypothesis

The intriguing role of module criticality in the generalization of deep networks

Federated Learning with Personalization Layers

Model Fusion via Optimal Transport

Deep Ensembles: A Loss Landscape Perspective

Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks

Bad Global Minima Exist and SGD Can Reach Them

Are All Layers Created Equal?

Moment Matching for Multi-Source Domain Adaptation

Efficient Decentralized Deep Learning by Dynamic Model Averaging

Robustness May Be at Odds with Accuracy

Averaging Weights Leads to Wider Optima and Better Generalization

Essentially No Barriers in Neural Network Energy Landscape

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

A Walk with SGD

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

Robustness and generalization

On the Surprising Behavior of Distance Metrics in High Dimensional Spaces

Which Layer is Learning Faster? A Systematic Exploration of Layer-wise Convergence Rate for Deep Neural Networks

On Convexity and Linear Mode Connectivity in Neural Networks

How SGD Selects the Global Minima in Over-parameterized Learning : A Dynamical Stability Perspective

Layerwise interpolations (left) and robustness to random perturbations of the same norm (right) for vision transformers trained on CIFAR-10 with different learning rates

(a) α = 0 . 5 (b) α = 1 . 0 Figure 29:

A suite for analyzing large

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names