Wasserstein Barycenter-based Model Fusion and Linear Mode Connectivity of Neural Networks (2022-10-13T00:00:00.000000Z)

TL;DR

A unified mathematical framework for neural network (NN) model fusion is proposed and utilized to reveal new insights about the linear mode connectivity of SGD solutions and to provide new empirical evidence for recent conjectures.

Abstract

Based on the concepts of Wasserstein barycenter (WB) and Gromov-Wasserstein barycenter (GWB), we propose a unified mathematical framework for neural network (NN) model fusion and utilize it to reveal new insights about the linear mode connectivity of SGD solutions. In our framework, the fusion occurs in a layer-wise manner and builds on an interpretation of a node in a network as a function of the layer preceding it. The versatility of our mathematical framework allows us to talk about model fusion and linear mode connectivity for a broad class of NNs, including fully connected NN, CNN, ResNet, RNN, and LSTM, in each case exploiting the specific structure of the network architecture. We present extensive numerical experiments to: 1) illustrate the strengths of our approach in relation to other model fusion methodologies and 2) from a certain perspective, provide new empirical evidence for recent conjectures which say that two local minima found by gradient-based methods end up lying on the same basin of the loss landscape after a proper permutation of weights is applied to one of the models.

Authors

Aditya Kumar Akash

1 papers

Sixu Li

1 papers

N. G. Trillos

1 papers

TL;DR

Abstract

Authors

References63 items

Random initialisations performing above chance and how to find them

Git Re-Basin: Merging Models modulo Permutation Symmetries

Federated Learning with Position-Aware Neurons

The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

Fed2: Feature-Aligned Federated Learning

Understanding deep learning (still) requires rethinking generalization

FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning

What is being transferred in transfer learning?

Model Fusion with Kullback-Leibler Divergence

Federated Learning with Matched Averaging

Advances and Open Problems in Federated Learning

Model Fusion via Optimal Transport

Bayesian Nonparametric Federated Learning of Neural Networks

Scalable Gromov-Wasserstein Learning for Graph Partitioning and Matching

Wasserstein Barycenter Model Ensembling

Gromov-Wasserstein Learning for Graph Matching and Node Embedding

On the loss landscape of a class of deep neural networks with no bad local valleys

Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced

A mean field view of the landscape of two-layer neural networks

Averaging Weights Leads to Wider Optima and Better Generalization

Optimal Transport for Multi-source Domain Adaptation under Target Shift

Essentially No Barriers in Neural Network Energy Landscape

Computational Optimal Transport

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

Visualizing the Loss Landscape of Neural Nets

Exploring Generalization in Deep Learning

Multilevel Clustering via Wasserstein Means

A Transportation Lp\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L^p$$\end{document} Distance for Signal Analysis

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

Gromov-Wasserstein Averaging of Kernel and Distance Matrices

Communication-Efficient Learning of Deep Networks from Decentralized Data

Deep Residual Learning for Image Recognition

Character-level Convolutional Networks for Text Classification

Scalable Bayes via Barycenter in Wasserstein Space

Discrete Wasserstein barycenters: optimal transport for discrete data

WASP: Scalable Bayes via barycenters of subset posteriors

Adam: A Method for Stochastic Optimization

Iterative Bregman Projections for Regularized Transportation Problems

Optimal Transport for Domain Adaptation

GloVe: Global Vectors for Word Representation

Very Deep Convolutional Networks for Large-Scale Image Recognition

Continuum Limit of Total Variation on Point Clouds

Fast Computation of Wasserstein Barycenters

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Sinkhorn Distances: Lightspeed Computation of Optimal Transport

The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]

Gromov–Wasserstein Distances and the Metric Approach to Object Matching

Barycenters in the Wasserstein Space

Optimal Transport: Old and New

Hierarchical Beta Processes and the Indian Buffet Process

Diagonal equivalence to matrices with prescribed row and column sums. II

Deep Neural Network Fusion via Graph Matching with Applications to Model Ensemble and Federated Learning

2022) solve the alignment problem during the training process of local models

Model Fusion of Heterogeneous Neural Networks via Cross-Layer Alignment

butions and the output ensemble is used to generate the global model using knowledge distillation

We define the TL space associated to X (see Garcı́a Trillos & Slepčev

2016), a stationary point of this optimization problem can be reached following the iterations: Πl

GROMOV-WASSERSTEIN DISTANCE AND GROMOV-WASSERSTEIN BARYCENTERS The Gromov-Wasserstein problem Mémoli

Optimal Transport for Applied Mathematicians

A RELATED WORKS A.1 OTHER MODEL FUSION ALGORITHMS Different from the post-processing strategies proposed in Singh & Jaggi (2019)

2015) extended the entropic regularization idea to the barycenter problem and proposed to solve it through sub-gradient descent and Bregman projection iterations

Learning Multiple Layers of Features from Tiny Images

A Note Concerning Simultaneous Integral Equations

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names