Safety and Performance, Why not Both? Bi-Objective Optimized Model Compression toward AI Software Deployment (2022-08-11T00:00:00.000000Z)

TL;DR

A concrete safe model compression mechanism, called MIA-SafeCompress, which can automatically compress a big model to a small one following the dynamic sparse training paradigm, inspired by the test-driven development (TDD) paradigm in software engineering is proposed.

Abstract

The size of deep learning models in artificial intelligence (AI) software is increasing rapidly, which hinders the large-scale deployment on resource-restricted devices (e.g., smartphones). To mitigate this issue, AI software compression plays a crucial role, which aims to compress model size while keeping high performance. However, the intrinsic defects in the big model may be inherited by the compressed one. Such defects may be easily leveraged by attackers, since the compressed models are usually deployed in a large number of devices without adequate protection. In this paper, we try to address the safe model compression problem from a safety-performance co-optimization perspective. Specifically, inspired by the test-driven development (TDD) paradigm in software engineering, we propose a test-driven sparse training framework called SafeCompress. By simulating the attack mechanism as the safety test, SafeCompress can automatically compress a big model to a small one following the dynamic sparse training paradigm. Further, considering a representative attack, i.e., membership inference attack (MIA), we develop a concrete safe model compression mechanism, called MIA-SafeCompress. Extensive experiments are conducted to evaluate MIA-SafeCompress on five datasets for both computer vision and natural language processing tasks. The results verify the effectiveness and generalization of our method. We also discuss how to adapt SafeCompress to other attacks besides MIA, demonstrating the flexibility of SafeCompress.

Authors

Jie Zhu

1 papers

Leye Wang

1 papers

Xiao Han

1 papers

TL;DR

Abstract

Authors

References70 items

ReMoS: Reducing Defect Inheritance in Transfer Learning via Relevant Model Slicing

Membership Inference Attacks and Defenses in Neural Network Pruning

Maximum Entropy on Erroneous Predictions (MEEP): Improving model calibration for medical image segmentation

Scaling Vision with Sparse Mixture of Experts

Chasing Sparsity in Vision Transformers: An End-to-End Exploration

Top-KAST: Top-K Always Sparse Training

Membership Privacy for Machine Learning Models Through Knowledge Transfer

Distilling Knowledge from Publicly Available Online EMR Data to Emerging Epidemic for Prognosis

ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models

Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Against Membership Inference Attack: Pruning is All You Need

Ensemble Learning-Based Technique for Force Classifications in Piezoelectric Touch Panels

How Does Data Augmentation Affect Privacy in Machine Learning?

Language Models are Few-Shot Learners

Systematic Evaluation of Privacy Risks of Machine Learning Models

Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey

Sparse Weight Activation Training

Rigging the Lottery: Making All Tickets Winners

F-PAD: Private Attribute Disclosure Risk Estimation in Online Social Networks

TinyBERT: Distilling BERT for Natural Language Understanding

MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examples

RoBERTa: A Robustly Optimized BERT Pretraining Approach

SciPy 1.0: fundamental algorithms for scientific computing in Python

Sparse Networks from Scratch: Faster Training without Losing Performance

White-box vs Black-box: Bayes Optimal Strategies for Membership Inference

Evaluating Differentially Private Machine Learning in Practice

Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization

Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware

Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

Machine Learning with Membership Privacy using Adversarial Regularization

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

Moonshine: Distilling with Cheap Convolutions

NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm

Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting

Learning Efficient Convolutional Networks through Network Slimming

Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science

Network computations in artificial intelligence

Regularizing Neural Networks by Penalizing Confident Output Distributions

Membership Inference Attacks Against Machine Learning Models

Deep Learning with Differential Privacy

Ternary Weight Networks

Deep Residual Learning for Image Recognition

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Character-level Convolutional Networks for Text Classification

Learning both Weights and Connections for Efficient Neural Network

Compressing Neural Networks with the Hashing Trick

Distilling the Knowledge in a Neural Network

Very Deep Convolutional Networks for Large-Scale Image Recognition

Do Deep Nets Really Need to be Deep?

Private traits and attributes are predictable from digital records of human behavior

ImageNet classification with deep convolutional neural networks

Representation Learning: A Review and New Perspectives

Test Driven Development: By Example

Compressing Java class files

Code compression

Executing Compressed Programs On An Embedded RISC Architecture

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Resisting membership inference attacks through knowledge distillation

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Roberta: A Safety and Performance, Why not Both? Bi-Objective Optimized Model Compression toward AI Software DeploymentASE ’22

Membership Inference Attack against Differentially Private Deep Learning Model

Binarized Neural Networks

Ran El-Yaniv, and Yoshua Bengio

Tiny ImageNet Visual Recognition Challenge

Dropout: a simple way to prevent neural networks from overfitting

Learning Multiple Layers of Features from Tiny Images

PPMexe: Program compression

Field of Study

Journal Information

Name

Venue Information

Name

Type