CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (2021-02-09T00:00:00.000000Z)

TL;DR

This paper introduces CodeXGLUE, a benchmark dataset to foster machine learning research for program understanding and generation that includes a collection of 10 tasks across 14 datasets and a platform for model evaluation and comparison.

Abstract

Benchmark datasets have a significant impact on accelerating research in programming language tasks. In this paper, we introduce CodeXGLUE, a benchmark dataset to foster machine learning research for program understanding and generation. CodeXGLUE includes a collection of 10 tasks across 14 datasets and a platform for model evaluation and comparison. CodeXGLUE also features three baseline systems, including the BERT-style, GPT-style, and Encoder-Decoder models, to make it easy for researchers to use the platform. The availability of such data and baselines can help the development and validation of new methods that can be applied to various program understanding and generation problems.

Authors

Shuai Lu

5 papers

Daya Guo

5 papers

Shuo Ren

4 papers

TL;DR

Abstract

Authors

References117 items

CoSQA: 20,000+ Web Queries for Code Search and Question Answering

PyMT5: Multi-mode Translation of Natural Language and Python Code with Transformers

CodeBLEU: a Method for Automatic Evaluation of Code Synthesis

GraphCodeBERT: Pre-training Code Representations with Data Flow

Unit Test Case Generation with Transformers

Semantic code search via equational reasoning

Unsupervised Translation of Programming Languages

MISIM: An End-to-End Neural Code Similarity System

CoCoGUM: Contextual Code Summarization with Multi-Relational GNN on UMLs

IntelliCode compose: code generation using transformer

Semantic Scaffolds for Pseudocode-to-Code Generation

Global Relational Models of Source Code

Incorporating External Knowledge through Pre-training for Natural Language to Code Generation

XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation

XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization

Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code

TranS^3: A Transformer-based Framework for Unifying Code Summarization and Code Search

CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Detecting Code Clones with Graph Neural Network and Flow-Augmented Abstract Syntax Tree

Are the Code Snippets What We Are Searching for? A Benchmark and an Empirical Study on Code Search with Natural-Language Queries

oLMpics-On What Language Model Pre-training Captures

Unsupervised Cross-lingual Representation Learning at Scale

Improving bug detection via context-based code representation learning and attention-based neural networks

Code Generation as a Dual Task of Code Summarization

CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks

Language Models as Knowledge Bases?

Release Strategies and the Social Impacts of Language Models

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Pythia: AI-assisted Code Completion System

SPoC: Search-based Pseudocode to Code

Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

Neural Detection of Semantic Code Clones Via Tree-Based Convolution

A Novel Neural Source Code Representation Based on Abstract Syntax Tree

Learning Programmatic Idioms for Scalable Semantic Parsing

Neural Program Repair by Jointly Learning to Localize and Repair

Learning-Based Recursive Aggregation of Abstract Syntax Trees for Code Clone Detection

An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation

Aroma: code recommendation via structural code search

Structured Neural Summarization

Improving Automatic Source Code Summarization via Deep Reinforcement Learning

Mapping Language to Code in Programmatic Context

Summarizing Source Code with Transferred API Knowledge

Neural Code Comprehension: A Learnable Representation of Code Semantics

Deep Code Search

Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow

DeepBugs: a learning approach to name-based bug detection

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

code2vec: learning distributed representations of code

StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow

Tree-to-tree Neural Networks for Program Translation

Learning to Represent Programs with Graphs

A Survey of Machine Learning for Big Code and Naturalness

Are deep neural networks the best choice for modeling source code?

Supervised Deep Features for Software Functional Clone Detection by Exploiting Lexical and Syntactical Information in Source Code

Program Synthesis

Attention is All you Need

Bayesian Sketch Learning for Program Synthesis

DeepFix: Fixing Common C Language Errors by Deep Learning

Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

Probabilistic model for code with decision trees

Deep learning code fragments for code clone detection

Bugram: Bug detection with n-gram language models

Summarizing Source Code using a Neural Attention Model

PHOG: Probabilistic Model for Code

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Automatically Learning Semantic Features for Defect Prediction

Refinement types for TypeScript

A Convolutional Attention Network for Extreme Summarization of Source Code

Neural Programmer-Interpreters

Neural Programmer: Inducing Latent Programs with Gradient Descent

Divide-and-Conquer Approach for Multi-phase Statistical Migration for Source Code (T)

Neural Machine Translation of Rare Words with Subword Units

Predicting a Correct Program in Programming by Example

On the "Naturalness" of Buggy Code