Contrastive Code Representation Learning

Published in

Conference on Empirical Methods in Natural Lang...(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

Contracode is proposed: a contrastive pre-training task that learns code functionality, not form, and improves summarization and TypeScript type inference accuracy by 2 to 13 percentage points over competitive baselines.

Abstract

Recent work learns contextual representations of source code by reconstructing tokens from their context. For downstream semantic understanding tasks like code clone detection, these representations should ideally capture program functionality. However, we show that the popular reconstruction-based RoBERTa model is sensitive to source code edits, even when the edits preserve semantics. We propose ContraCode: a contrastive pre-training task that learns code functionality, not form. ContraCode pre-trains a neural network to identify functionally similar variants of a program among many non-equivalent distractors. We scalably generate these variants using an automated source-to-source compiler as a form of data augmentation. Contrastive pre-training outperforms RoBERTa on an adversarial code clone detection benchmark by 39% AUROC. Surprisingly, improved adversarial robustness translates to better accuracy over natural code; ContraCode improves summarization and TypeScript type inference accuracy by 2 to 13 percentage points over competitive baselines. All source is available at https://github.com/parasj/contracode.

Authors

P. Abbeel

35 papers

Joseph E. Gonzalez

10 papers

Ion Stoica

5 papers

References95 items

Evaluating Large Language Models Trained on Code

Scaling Laws for Transfer

You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion

Debiased Contrastive Learning

DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations

Contrastive Code Representation Learning

Published in

Conference on Empirical Methods in Natural Lang...(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

P. Abbeel

35 papers

Joseph E. Gonzalez

10 papers

Ion Stoica

5 papers

References95 items

Evaluating Large Language Models Trained on Code

Scaling Laws for Transfer

You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion

Debiased Contrastive Learning

DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations

Tianjun Zhang

6 papers

On Mutual Information in Contrastive Learning for Visual Representations

A Transformer-based Approach for Source Code Summarization

LambdaNet: Probabilistic Type Inference using Graph Neural Networks

SCELMo: Source Code Embeddings from Language Models

Evaluation of Generalizability of Neural Program Analyzers under Semantic-Preserving Transformations

Typilus: neural type hints

OptTyper: Probabilistic Type Inference by Optimising Logical and Natural Constraints

Improved Baselines with Momentum Contrastive Learning

CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Recommendation of Move Method Refactoring Using Path-Based Representation of Code

A Simple Framework for Contrastive Learning of Visual Representations

Adversarial Robustness for Code

Synthetic Datasets for Neural Program Synthesis

TypeWriter: neural type prediction with search-based validation

Momentum Contrast for Unsupervised Visual Representation Learning

Adversarial examples for models of code

Deep Transfer Learning for Source Code Modeling

Pre-trained Contextual Embedding of Source Code

CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

Visualizing and Understanding the Effectiveness of BERT

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Mercem: Method Name Recommendation Based on Call Graph Embedding

Learning to optimize halide with tree search and random programs

Learning Blended, Precise Semantic Program Embeddings

Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty

Contrastive Multiview Coding

COSET: A Benchmark for Evaluating Neural Program Embeddings

Data-Efficient Image Recognition with Contrastive Predictive Coding

Adversarial Examples Are Not Bugs, They Are Features

Defexts: A Curated Dataset of Reproducible Real-World Bugs for Modern JVM Languages

A Theoretical Analysis of Contrastive Unsupervised Representation Learning

The adverse effects of code duplication in machine learning models of code

Deep learning type inference

A Public Unified Bug Dataset for Java

Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks

code2seq: Generating Sequences from Structured Representations of Code

Representation Learning with Contrastive Predictive Coding

Neural Code Comprehension: A Learnable Representation of Code Semantics

Unsupervised Feature Learning via Non-parametric Instance Discrimination

Exploring the Limits of Weakly Supervised Pretraining

DeepBugs: a learning approach to name-based bug detection

Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

code2vec: learning distributed representations of code

Deep Contextualized Word Representations

Unsupervised Representation Learning by Predicting Image Rotations

Universal Language Model Fine-tuning for Text Classification

A Survey of Machine Learning for Big Code and Naturalness

One tool, many languages: language-parametric transformation with incremental parametric syntax

Challenges in Data-to-Document Generation

Device Placement Optimization with Reinforcement Learning

Attention is All you Need

Deep learning code fragments for code clone detection

Summarizing Source Code using a Neural Attention Model

Context Encoders: Feature Learning by Inpainting

Colorful Image Colorization

A Convolutional Attention Network for Extreme Summarization of Source Code

Neural Machine Translation of Rare Words with Subword Units

Bidirectional LSTM-CRF Models for Sequence Tagging

FaceNet: A unified embedding for face recognition and clustering

Towards a Big Data Curated Benchmark of Inter-project Code Clones

Understanding TypeScript

Distributed Representations of Words and Phrases and their Compositionality

Natural Language Models for Predicting Programming Comments

A field study of refactoring challenges and benefits

Scikit-learn: Machine Learning in Python

Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

Why Does Unsupervised Pre-training Help Deep Learning?

Automatic generation of peephole superoptimizers

Dimensionality Reduction by Learning an Invariant Mapping

Case studies in the use of ROC curve analysis for sensor-based estimates in human computer interaction

Programming Languages and Operating Systems

Denali: a goal-directed superoptimizer

Bidirectional recurrent neural networks

Signature Verification Using A "Siamese" Time Delay Neural Network

An investigation of the Therac-25 accidents

Superoptimizer: a look at the smallest program

“Cloze Procedure”: A New Tool for Measuring Readability

Classes of recursively enumerable sets and their decision problems

Babel: compiler for writing next generation javascript

The future of tech, with Kevin Scott and guests

CERT: Contrastive Self-supervised Learning for Language Understanding

Terser: Javascript parser, mangler and compressor toolkit for es6+

Learned TPU Cost Model for XLA Tensor Programs

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Phitchaya Mangpo Phothilimthana, and Mike Burrows

Improving Language Understanding by Generative Pre-Training

Technical support to the national highway traffic safety administration (nhtsa) on the reported toyota motor corporation (tmc) unintended acceleration (ua) investigation