Attribution Preservation in Network Compression for Reliable Network Interpretation (2020-10-28T00:00:00.000000Z)

TL;DR

This paper presents a framework that can preserve the attributions while compressing a network by employing the Weighted Collapsed Attribution Matching regularizer, and demonstrates the effectiveness of the algorithm both quantitatively and qualitatively on diverse compression methods.

Abstract

Neural networks embedded in safety-sensitive applications such as self-driving cars and wearable health monitors rely on two important techniques: input attribution for hindsight analysis and network compression to reduce its size for edge-computing. In this paper, we show that these seemingly unrelated techniques conflict with each other as network compression deforms the produced attributions, which could lead to dire consequences for mission-critical applications. This phenomenon arises due to the fact that conventional network compression methods only preserve the predictions of the network while ignoring the quality of the attributions. To combat the attribution inconsistency problem, we present a framework that can preserve the attributions while compressing a network. By employing the Weighted Collapsed Attribution Matching regularizer, we match the attribution maps of the network being compressed to its pre-compression former self. We demonstrate the effectiveness of our algorithm both quantitatively and qualitatively on diverse compression methods.

Authors

Eunho Yang

2 papers

Sung Ju Hwang

8 papers

Geondo Park

1 papers

TL;DR

Abstract

Authors

References35 items

Prediction of Radiation Pneumonitis With Dose Distribution: A Convolutional Neural Network (CNN) Based Model

Current status and future trends of clinical diagnoses via image-based deep learning

On the Efficacy of Knowledge Distillation

Learning What and Where to Transfer

Variational Information Distillation for Knowledge Transfer

Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networks

A Benchmark for Interpretability Methods in Deep Neural Networks

RISE: Randomized Input Sampling for Explanation of Black-box Models

Adversarial Dropout Regularization

Automatic differentiation in PyTorch

Methods for interpreting and understanding deep neural networks

Overcoming Catastrophic Forgetting by Incremental Moment Matching

Axiomatic Attribution for Deep Networks

Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

Pruning Filters for Efficient ConvNets

Learning Structured Sparsity in Deep Neural Networks

Top-Down Neural Attention by Excitation Backprop

Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures

Learning Deep Features for Discriminative Localization

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Learning both Weights and Connections for Efficient Neural Network

Variational Dropout and the Local Reparameterization Trick

Distilling the Knowledge in a Neural Network

Striving for Simplicity: The All Convolutional Net

FitNets: Hints for Thin Deep Nets

ImageNet Large Scale Visual Recognition Challenge

Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

Segmentation Propagation in ImageNet

Optimal Brain Surgeon: Extensions and performance comparisons

Dropout: a simple way to prevent neural networks from overfitting

Visualizing Higher-Layer Features of a Deep Network

The Pascal Visual Object Classes Challenge 2006 ( VOC 2006 ) Results

Optimal Brain Damage

For data augmentation, we use random resized crop and random horizontal flip provided by Torchvision and Pytorch

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names