A Unified Algebraic Perspective on Lipschitz Neural Networks (2023-03-06T00:00:00.000000Z)

TL;DR

A novel algebraic perspective unifies various types of 1-Lipschitz neural networks, including the ones previously mentioned, along with methods based on orthogonality and spectral methods, and shows that SLLs outperform previous approaches on certified robust accuracy.

Abstract

Important research efforts have focused on the design and training of neural networks with a controlled Lipschitz constant. The goal is to increase and sometimes guarantee the robustness against adversarial attacks. Recent promising techniques draw inspirations from different backgrounds to design 1-Lipschitz neural networks, just to name a few: convex potential layers derive from the discretization of continuous dynamical systems, Almost-Orthogonal-Layer proposes a tailored method for matrix rescaling. However, it is today important to consider the recent and promising contributions in the field under a common theoretical lens to better design new and improved layers. This paper introduces a novel algebraic perspective unifying various types of 1-Lipschitz neural networks, including the ones previously mentioned, along with methods based on orthogonality and spectral methods. Interestingly, we show that many existing techniques can be derived and generalized via finding analytical solutions of a common semidefinite programming (SDP) condition. We also prove that AOL biases the scaled weight to the ones which are close to the set of orthogonal matrices in a certain mathematical manner. Moreover, our algebraic condition, combined with the Gershgorin circle theorem, readily leads to new and diverse parameterizations for 1-Lipschitz network layers. Our approach, called SDP-based Lipschitz Layers (SLL), allows us to design non-trivial yet efficient generalization of convex potential layers. Finally, the comprehensive set of experiments on image classification shows that SLLs outperform previous approaches on certified robust accuracy. Code is available at https://github.com/araujoalexandre/Lipschitz-SLL-Networks.

Authors

A. Allauzen

2 papers

Alexandre Araujo

1 papers

Aaron J. Havens

1 papers

TL;DR

Abstract

Authors

References38 items

A Recipe for Improved Certifiable Robustness: Capacity and Data

Improved techniques for deterministic l2 robustness

LOT: Layer-wise Orthogonal Training on Improving l2 Certified Robustness

Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networks

A Quantitative Geometric Approach to Neural Network Smoothness

Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds

A Dynamical System Perspective for Lipschitz Neural Networks

Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100

Skew Orthogonal Convolutions

Orthogonalizing Convolutional Layers with the Cayley Transform

Globally-Robust Neural Networks

Lipschitz Bounded Equilibrium Networks

Training Robust Neural Networks Using Lipschitz Bounds

Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks

Randomized Smoothing of All Shapes and Sizes

Curse of Dimensionality on Randomized Smoothing for Certifiable Robustness

Square Attack: a query-efficient black-box adversarial attack via random search

Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks

Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attack

Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks

Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers

Safety Verification and Robustness Analysis of Neural Networks via Quadratic Constraints and Semidefinite Programming

Certified Adversarial Robustness via Randomized Smoothing

Semidefinite relaxations for certifying robustness to adversarial examples

Generalizable Adversarial Training via Spectral Normalization

Sorting out Lipschitz function approximation

Spectral Normalization for Generative Adversarial Networks

Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks

Provable defenses against adversarial examples via the convex outer adversarial polytope

Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation

Adam: A Method for Stochastic Optimization

The conference paper

System analysis via integral quadratic constraints

Matrix analysis

Cones of diagonally dominant matrices

Constructing Orthogonal Convolutions in an Explicit Manner

On the theory of stability of control systems

h ( x ) = x − 2 W diag   n X j =1 | W T W | ij q j + ε q i + ε   − 1 σ ( W T x + b )

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names