Evaluation of Generalizability of Neural Program Analyzers under Semantic-Preserving Transformations (2020-04-15T00:00:00.000000Z)

TL;DR

A large-scale evaluation of the generalizability of two popular neural program analyzers using seven semantically-equivalent transformations of programs to provide the initial stepping stones for quantifying robustness in neural program Analyzers.

Abstract

The abundance of publicly available source code repositories, in conjunction with the advances in neural networks, has enabled data-driven approaches to program analysis. These approaches, called neural program analyzers, use neural networks to extract patterns in the programs for tasks ranging from development productivity to program reasoning. Despite the growing popularity of neural program analyzers, the extent to which their results are generalizable is unknown. In this paper, we perform a large-scale evaluation of the generalizability of two popular neural program analyzers using seven semantically-equivalent transformations of programs. Our results caution that in many cases the neural program analyzers fail to generalize well, sometimes to programs with negligible textual differences. The results provide the initial stepping stones for quantifying robustness in neural program analyzers.

Authors

Mohammad Amin Alipour

7 papers

Md Rafiqul Islam Rabin

7 papers

TL;DR

Abstract

Authors

References41 items

Hoppity: Learning Graph Transformations to Detect and Fix Bugs in Programs

AutoFocus: Interpreting Attention-Based Neural Networks by Code Perturbation

Assessing the Generalizability of Code2vec Token Embeddings

Adversarial examples for models of code

Testing Neural Program Analyzers

Learning a Static Bug Finder from Data

Learning Blended, Precise Semantic Program Embeddings

COSET: A Benchmark for Evaluating Neural Program Embeddings

Learning Scalable and Precise Representation of Program Semantics

Maybe Deep Neural Networks are the Best Choice for Modeling Source Code

The adverse effects of code duplication in machine learning models of code

Deep learning type inference

code2seq: Generating Sequences from Structured Representations of Code

Adversarial Attack on Graph Structured Data

Adversarial Attacks on Neural Networks for Graph Data

AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation

code2vec: learning distributed representations of code

Dynamic Neural Program Embedding for Program Repair

Learning to Represent Programs with Graphs

Generating Natural Adversarial Examples

Boosting Adversarial Attacks with Momentum

A Survey of Machine Learning for Big Code and Naturalness

Are deep neural networks the best choice for modeling source code?

Adversarial Examples for Evaluating Reading Comprehension Systems

Axiomatic Attribution for Deep Networks

DeepFix: Fixing Common C Language Errors by Deep Learning

Understanding Neural Networks through Representation Erasure

Adversarial Machine Learning at Scale

Safety Verification of Deep Neural Networks

Towards Evaluating the Robustness of Neural Networks

sk_p: a neural program corrector for MOOCs

“Why Should I Trust You?”: Explaining the Predictions of Any Classifier

DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks

Recurrent Convolutional Neural Networks for Text Classification

Explaining and Harnessing Adversarial Examples

Application of Deep Belief Networks for Natural Language Understanding

Structured Generative Models of Natural Source Code

Intriguing properties of neural networks

Multilayer feedforward networks are universal approximators

Towards Verification of Artificial Neural Networks

Deep Learning

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names