Code Prediction by Feeding Trees to Transformers (2020-03-30T00:00:00.000000Z)

TL;DR

This work shows that by making the Transformer architecture aware of the syntactic structure of code, it increases the margin by which a Transformer-based system outperforms previous systems, and advances the state-of-the-art in the accuracy of code prediction (next token prediction) used in autocomplete systems.

Abstract

Code prediction, more specifically autocomplete, has become an essential feature in modern IDEs. Autocomplete is more effective when the desired next token is at (or close to) the top of the list of potential completions offered by the IDE at cursor position. This is where the strength of the underlying machine learning system that produces a ranked order of potential completions comes into play. We advance the state-of-the-art in the accuracy of code prediction (next token prediction) used in autocomplete systems. Our work uses Transformers as the base neural architecture. We show that by making the Transformer architecture aware of the syntactic structure of code, we increase the margin by which a Transformer-based system outperforms previous systems. With this, it outperforms the accuracy of several state-of-the-art next token prediction systems by margins ranging from 14% to 18%. We present in the paper several ways of communicating the code structure to the Transformer, which is fundamentally built for processing sequence data. We provide a comprehensive experimental evaluation of our proposal, along with alternative design choices, on a standard Python dataset, as well as on Facebook internal Python corpus. Our code and data preparation pipeline will be available in open source.

Authors

Seohyun Kim

2 papers

Jinman Zhao

1 papers

Yuchi Tian

2 papers

TL;DR

Abstract

Authors

References70 items

Predicting Defective Lines Using a Model-Agnostic Technique

IntelliCode compose: code generation using transformer

Global Relational Models of Source Code

Fast and Memory-Efficient Neural Code Completion

Sequence Model Design for Code Completion in the Modern IDE

Adversarial Attacks on Deep-learning Models in Natural Language Processing

An Empirical Study of Model-Agnostic Techniques for Defect Prediction Models

Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code

Tree-structured Attention with Hierarchical Accumulation

Modeling Programs Hierarchically with Stack-Augmented LSTM

Semantic Robustness of Models of Source Code

Novel positional encodings to enable tree-based transformers

Adversarial examples for models of code

Structural Language Models of Code

Structural Language Models for Any-Code Generation

Tree Transformer: Integrating Tree Structures into Self-Attention

Tree-Transformer: A Transformer-Based Method for Correction of Tree-Structured Data

Machine Learning Interpretability: A Survey on Methods and Metrics

Pythia: AI-assisted Code Completion System

Improve Language Modelling for Code Completion through Learning General Token Repetition of Source Code

You Only Need Attention to Traverse Trees

COSET: A Benchmark for Evaluating Neural Program Embeddings

Language Modeling with Deep Transformers

Unified Language Model Pre-training for Natural Language Understanding and Generation

When Code Completion Fails: A Case Study on Real-World Completions

Neural Program Repair by Jointly Learning to Localize and Repair

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability

Adversarial Attacks and Defences: A Survey

Structured Neural Summarization

Open Vocabulary Learning on Source Code with a Graph-Structured Cache

Character-Level Language Modeling with Deeper Self-Attention

code2seq: Generating Sequences from Structured Representations of Code

Generative Code Modeling with Graphs

Building Language Models for Text with Named Entities

code2vec: learning distributed representations of code

Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey

Visual interpretability for deep learning: a survey

Code Completion with Neural Attention and Pointer Networks

Learning to Represent Programs with Graphs

A Survey of Machine Learning for Big Code and Naturalness

Are deep neural networks the best choice for modeling source code?

Interpretability of deep learning models: A survey of results

Attention is All you Need

SmoothGrad: removing noise by adding noise

Neural Code Completion

Axiomatic Attribution for Deep Networks

Program Synthesis for Character Level Language Modeling

Probabilistic model for code with decision trees

Collective Intelligence for Smarter API Recommendations in Python

Summarizing Source Code using a Neural Attention Model

PHOG: Probabilistic Model for Code

“Why Should I Trust You?”: Explaining the Predictions of Any Classifier

A Convolutional Attention Network for Extreme Summarization of Source Code

Learning programs from noisy data

Intelligent Code Completion with Bayesian Networks

Gated Graph Sequence Neural Networks

Pointer Networks

Code completion with statistical language models

Mining idioms from source code

Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

A statistical semantic language model for source code

On the naturalness of software

Long Short-Term Memory

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Autocompletion with deep learning

150k python dataset

Learning autocompletion from realworld datasets

Galois autocompleter

Field of Study

Journal Information

Name

Page

Venue Information

Name