0/1 Deep Neural Networks via Block Coordinate Descent

Published in

arXiv.org(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

This paper reformulates 0/1 DNNs as an unconstrained optimization problem and then solves it by a block coordinate descend (BCD) method, acquiring closed-form solutions for sub-problems of BCD as well as its convergence properties.

Abstract

The step function is one of the simplest and most natural activation functions for deep neural networks (DNNs). As it counts 1 for positive variables and 0 for others, its intrinsic characteristics (e.g., discontinuity and no viable information of subgradients) impede its development for several decades. Even if there is an impressive body of work on designing DNNs with continuous activation functions that can be deemed as surrogates of the step function, it is still in the possession of some advantageous properties, such as complete robustness to outliers and being capable of attaining the best learning-theoretic guarantee of predictive accuracy. Hence, in this paper, we aim to train DNNs with the step function used as an activation function (dubbed as 0/1 DNNs). We first reformulate 0/1 DNNs as an unconstrained optimization problem and then solve it by a block coordinate descend (BCD) method. Moreover, we acquire closed-form solutions for sub-problems of BCD as well as its convergence properties. Furthermore, we also integrate $\ell_{2,0}$-regularization into 0/1 DNN to accelerate the training process and compress the network scale. As a result, the proposed algorithm has a high performance on classifying MNIST and Fashion-MNIST datasets. As a result, the proposed algorithm has a desirable performance on classifying MNIST, FashionMNIST, Cifar10, and Cifar100 datasets.

Authors

Geoffrey Y. Li

4 papers

Hui Zhang

1 papers

Shenglong Zhou

1 papers

References59 items

Local Means Binary Networks for Image Super-Resolution

Toward Pixel-Level Precision for Binary Super-Resolution With Mixed Binary Representation

Toward Accurate Binarized Neural Networks With Sparsity for Mobile Application

Recursion Newton-Like Algorithm for l2,0-ReLU Deep Neural Networks

A comprehensive review of Binary Neural Network

0/1 Deep Neural Networks via Block Coordinate Descent

Published in

arXiv.org(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Geoffrey Y. Li

4 papers

Hui Zhang

1 papers

Shenglong Zhou

1 papers

References59 items

Local Means Binary Networks for Image Super-Resolution

Toward Pixel-Level Precision for Binary Super-Resolution With Mixed Binary Representation

Toward Accurate Binarized Neural Networks With Sparsity for Mobile Application

Recursion Newton-Like Algorithm for l2,0-ReLU Deep Neural Networks

A comprehensive review of Binary Neural Network

Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

Computing One-Bit Compressive Sensing via Double-Sparsity Constrained Optimization

Quadratic Convergence of Smoothing Newton's Method for 0/1 Loss Optimization

High-Capacity Expert Binary Networks

Distillation-Guided Residual Learning for Binary Convolutional Neural Networks

Newton method for ℓ0-regularized optimization

Training Binary Neural Networks with Real-to-Binary Convolutions

Sparsity Meets Robustness: Channel Pruning for the Feynman-Kac Formalism Principled Robust Deep Neural Nets

Toward Compact ConvNets via Structure-Sparsity Regularized Filter Pruning

Channel Pruning for Deep Neural Networks Via a Relaxed Groupwise Splitting Method

Accurate and Compact Convolutional Neural Networks with Trained Binarization

Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation

Quasi-hyperbolic momentum and Adam for deep learning

An Empirical study of Binary Neural Networks' Optimisation

Modulated Convolutional Networks

Optimization problems involving group sparsity terms

Aggregated Momentum: Stability Through Passive Damping

Global Convergence of Block Coordinate Descent in Deep Learning

On the Convergence of Adam and Beyond

A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Training

Decoupled Weight Decay Regularization

Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks

Swish: a Self-Gated Activation Function

Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

Self-Normalizing Neural Networks

How to Train a Compact Binary Neural Network with High Accuracy?

An overview of gradient descent optimization algorithms

Training Neural Networks Without Gradients: A Scalable ADMM Approach

Incorporating Nesterov Momentum into Adam

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

BinaryConnect: Training Deep Neural Networks with binary weights during propagations

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Adam: A Method for Stochastic Optimization

ADADELTA: An Adaptive Learning Rate Method

Sensitivity-Based Adaptive Learning Rules for Binary Feedforward Neural Networks

Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

Rectified Linear Units Improve Restricted Boltzmann Machines

All‐or‐None Law

The Influence of the Sigmoid Function Parameters on the Speed of Backpropagation Learning

Learning representations by back-propagating errors

Computing a Trust Region Step

“Machine learning”

“Support vector machine classifier via L 0 / 1 soft-margin loss,”

Robust Detection of Adversarial Attacks by Modeling the Intrinsic Properties of Deep Neural Networks

Group Sparse Optimization via lp, q Regularization

Variational Analysis

“The CIFAR-10 dataset,”

Rectifier Nonlinearities Improve Neural Network Acoustic Models

“Neural networks for machine learning lecture 6a overview of mini-batch gradient descent,”

“MNIST handwritten digit database,”

A logical calculus of the ideas immanent in nervous activity

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2206.09379

Venue Information

Name

arXiv.org

Type

URL

https://arxiv.org

Alternate Names

ArXiv

TL;DR

Abstract

Authors

References59 items

Local Means Binary Networks for Image Super-Resolution

Toward Pixel-Level Precision for Binary Super-Resolution With Mixed Binary Representation

Toward Accurate Binarized Neural Networks With Sparsity for Mobile Application

Recursion Newton-Like Algorithm for l2,0-ReLU Deep Neural Networks

A comprehensive review of Binary Neural Network

TL;DR

Abstract

Authors

References59 items

Local Means Binary Networks for Image Super-Resolution

Toward Pixel-Level Precision for Binary Super-Resolution With Mixed Binary Representation

Toward Accurate Binarized Neural Networks With Sparsity for Mobile Application

Recursion Newton-Like Algorithm for l2,0-ReLU Deep Neural Networks

A comprehensive review of Binary Neural Network

Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

Computing One-Bit Compressive Sensing via Double-Sparsity Constrained Optimization

Quadratic Convergence of Smoothing Newton's Method for 0/1 Loss Optimization

High-Capacity Expert Binary Networks

Distillation-Guided Residual Learning for Binary Convolutional Neural Networks

Newton method for ℓ0-regularized optimization

Training Binary Neural Networks with Real-to-Binary Convolutions

Sparsity Meets Robustness: Channel Pruning for the Feynman-Kac Formalism Principled Robust Deep Neural Nets

Toward Compact ConvNets via Structure-Sparsity Regularized Filter Pruning

Channel Pruning for Deep Neural Networks Via a Relaxed Groupwise Splitting Method

Accurate and Compact Convolutional Neural Networks with Trained Binarization

Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation

Quasi-hyperbolic momentum and Adam for deep learning

An Empirical study of Binary Neural Networks' Optimisation

Modulated Convolutional Networks

Optimization problems involving group sparsity terms

Aggregated Momentum: Stability Through Passive Damping

Global Convergence of Block Coordinate Descent in Deep Learning

On the Convergence of Adam and Beyond

A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Training

Decoupled Weight Decay Regularization

Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks

Swish: a Self-Gated Activation Function

Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

Self-Normalizing Neural Networks

How to Train a Compact Binary Neural Network with High Accuracy?

An overview of gradient descent optimization algorithms

Training Neural Networks Without Gradients: A Scalable ADMM Approach

Incorporating Nesterov Momentum into Adam

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

BinaryConnect: Training Deep Neural Networks with binary weights during propagations

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Adam: A Method for Stochastic Optimization

ADADELTA: An Adaptive Learning Rate Method

Sensitivity-Based Adaptive Learning Rules for Binary Feedforward Neural Networks

Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

Rectified Linear Units Improve Restricted Boltzmann Machines

All‐or‐None Law

The Influence of the Sigmoid Function Parameters on the Speed of Backpropagation Learning

Problem (1)

Learning representations by back-propagating errors

Computing a Trust Region Step

Learning

“Machine learning”

“Support vector machine classifier via L 0 / 1 soft-margin loss,”

Robust Detection of Adversarial Attacks by Modeling the Intrinsic Properties of Deep Neural Networks

Group Sparse Optimization via lp, q Regularization

Deep Learning

Variational Analysis

“The CIFAR-10 dataset,”

Rectifier Nonlinearities Improve Neural Network Acoustic Models

“Neural networks for machine learning lecture 6a overview of mini-batch gradient descent,”

“MNIST handwritten digit database,”

A logical calculus of the ideas immanent in nervous activity

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL