Memorization and Generalization in Neural Code Intelligence Models (2021-06-16T00:00:00.000000Z)

TL;DR

This work evaluates the memorization and generalization tendencies in neural code intelligence models through a case study across several benchmarks and model families by leveraging established approaches from other fields that use DNNs, such as introducing targeted noise into the training dataset.

Authors

Mohammad Amin Alipour

7 papers

Md Rafiqul Islam Rabin

7 papers

Aftab Hussain

2 papers

Vincent J. Hellendoorn

2 papers

TL;DR

Authors

References94 items

StartCards - A method for early-stage software startups

Syntax-guided program reduction for understanding neural code intelligence models

Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models

A systematic evaluation of large language models of code

Code2Snapshot: Using Code Snapshots for Learning Representations of Source Code

A Survey on Machine Learning Techniques for Source Code Analysis

Evaluating Large Language Models Trained on Code

Multimodal Representation for Neural Code Search

Understanding neural code intelligence through program simplification

Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜

CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

Memorization vs. Generalization : Quantifying Data Leakage in NLP Performance Evaluation

Probing model signal-awareness via prediction-preserving input minimization

Towards demystifying dimensions of source code embeddings

On the generalizability of Neural Program Models with respect to semantic-preserving program transformations

Measuring Memorization Effect in Word-Level Neural Networks Probing

Global Relational Models of Source Code

Hoppity: Learning Graph Transformations to Detect and Fix Bugs in Programs

Evaluation of Generalizability of Neural Program Analyzers under Semantic-Preserving Transformations

Embedding Java Classes with code2vec: Improvements from Variable Obfuscation

Improved Code Summarization via a Graph Neural Network

CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Deep Learning for Source Code Modeling and Generation

Adversarial Robustness for Code

Semantic Robustness of Models of Source Code

Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study

Generalization through Memorization: Nearest Neighbor Language Models

Assessing the Generalizability of Code2vec Token Embeddings

Adversarial examples for models of code

CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

Testing Neural Program Analyzers

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Machine Learning Testing: Survey, Landscapes and Horizons

How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset

COSET: A Benchmark for Evaluating Neural Program Embeddings

Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasets

Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels

Learning to Spot and Refactor Inconsistent Method Names

Neural Program Repair by Jointly Learning to Localize and Repair

Identity Crisis: Memorization and Generalization under Extreme Overparameterization

Do ImageNet Classifiers Generalize to ImageNet?

Bilateral Dependency Neural Networks for Cross-Language Algorithm Classification

The adverse effects of code duplication in machine learning models of code

Learning to Learn From Noisy Labeled Data

Deep learning type inference

Prediction of relatedness in stack overflow: deep learning vs. SVM: a reproducibility study

Improving Automatic Source Code Summarization via Deep Reinforcement Learning

code2seq: Generating Sequences from Structured Representations of Code

Know What You Don’t Know: Unanswerable Questions for SQuAD

Studying the difference between natural and programming language corpora

Insights on representational similarity in neural networks with canonical correlation

Deep Code Search

Learning to Attack: Adversarial Transformation Networks

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Adversarially Robust Generalization Requires More Data

code2vec: learning distributed representations of code

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

Learning to Represent Programs with Graphs

DéjàVu: a map of code duplicates on GitHub

A Survey of Machine Learning for Big Code and Naturalness

A Closer Look at Memorization in Deep Networks

Attention is All you Need

Curating GitHub for engineered software projects

What do Neural Machine Translation Models Learn about Morphology?

Understanding deep learning requires rethinking generalization

Probabilistic model for code with decision trees

Summarizing Source Code using a Neural Attention Model

Learning Unified Features from Natural and Programming Languages for Locating Buggy Source Code

Deep API learning

Identity Mappings in Deep Residual Networks

A Convolutional Attention Network for Extreme Summarization of Source Code

Google AI algorithm masters ancient game of Go

Learning programs from noisy data

Gated Graph Sequence Neural Networks

Suggesting accurate method and class names

Toward Deep Learning Software Repositories