InCoder: A Generative Model for Code Infilling and Synthesis (2022-04-12T00:00:00.000000Z)

TL;DR

InCoder is introduced, a unified generative model that can perform program synthesis (via left-to-right generation) as well as editing (via infilling) with the ability to condition on bidirectional context substantially improves performance on these tasks, while still performing comparably on standard program synthesis benchmarks.

Abstract

Code is seldom written in a single left-to-right pass and is instead repeatedly edited and refined. We introduce InCoder, a unified generative model that can perform program synthesis (via left-to-right generation) as well as editing (via infilling). InCoder is trained to generate code files from a large corpus of permissively licensed code, where regions of code have been randomly masked and moved to the end of each file, allowing code infilling with bidirectional context. Our model is the first generative model that is able to directly perform zero-shot code infilling, which we evaluate on challenging tasks such as type inference, comment generation, and variable re-naming. We find that the ability to condition on bidirectional context substantially improves performance on these tasks, while still performing comparably on standard program synthesis benchmarks in comparison to left-to-right only models pretrained at similar scale. The InCoder models and code are publicly released. https://sites.google.com/view/incoder-code-models

Authors

Sida I. Wang

4 papers

Wen-tau Yih

5 papers

Luke Zettlemoyer

46 papers

TL;DR

Abstract

Authors

References81 items

Efficient Training of Language Models to Fill in the Middle

GPT-NeoX-20B: An Open-Source Autoregressive Language Model

PaLM: Scaling Language Modeling with Pathways

Training language models to follow instructions with human feedback

A systematic evaluation of large language models of code

Deduplicating Training Data Mitigates Privacy Risks in Language Models

CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences

Competition-level code generation with AlphaCode

LaMDA: Language Models for Dialog Applications

CM3: A Causal Masked Multimodal Model of the Internet

Efficient Large Scale Language Modeling with Mixtures of Experts

Finetuned Language Models Are Zero-Shot Learners

CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

Learning type annotation: is big data enough?

Program Synthesis with Large Language Models

HTLM: Hyper-Text Pre-Training and Prompting of Language Models

Deduplicating Training Data Makes Language Models Better

Evaluating Large Language Models Trained on Code

SpreadsheetCoder: Formula Prediction from Semi-structured Context

Break-It-Fix-It: Unsupervised Learning for Program Repair

Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections

Unified Pre-training for Program Understanding and Generation

DOBF: A Deobfuscation Pre-Training Objective for Programming Languages

CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

Scaling Laws for Autoregressive Generative Modeling

Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models

PyMT5: Multi-mode Translation of Natural Language and Python Code with Transformers

Language Models are Few-Shot Learners

Graph-based, Self-Supervised Program Repair from Diagnostic Feedback

Enabling Language Models to Fill in the Blanks

LambdaNet: Probabilistic Type Inference using Graph Neural Networks

Typilus: neural type hints

CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Scaling Laws for Neural Language Models

Learning and Evaluating Contextual Embedding of Source Code

TypeWriter: neural type prediction with search-based validation

PyTorch: An Imperative Style, High-Performance Deep Learning Library

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

XL-Editor: Post-editing Sentences with XLNet

Restoring ancient text using deep learning: a case study on Greek epigraphy

AutoPandas: neural-backed generators for program synthesis

JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation

Structural Language Models of Code

CLN2INV: Learning Loop Invariants with Continuous Logic Networks

CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

CTRL: A Conditional Transformer Language Model for Controllable Generation

SPoC: Search-based Pseudocode to Code

KERMIT: Generative Insertion-Based Modeling for Sequences

Defending Against Neural Fake News

The Curious Case of Neural Text Degeneration

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Mask-Predict: Parallel Decoding of Conditional Masked Language Models

Insertion Transformer: Flexible Sequence Generation via Insertion Operations

SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair

The adverse effects of code duplication in machine learning models of code

Deep learning type inference

Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task

Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow

Program Synthesis

Attention is All you Need

DeepFix: Fixing Common C Language Errors by Deep Learning

DeepCoder: Learning to Write Programs

Python probabilistic type inference with natural language support

Neural Machine Translation of Rare Words with Subword Units

Adam: A Method for Stochastic Optimization

EvoSuite: automatic test suite generation for object-oriented software

Automating string processing in spreadsheets using input-output examples

Combinatorial sketching for finite programs

Learning to Parse Database Queries Using Inductive Logic Programming

Space/time trade-offs in hash coding with allowable errors

A Conversational Paradigm for Program Synthesis

2022): we remove files that contain any line longer than 3000 tokens or an average line length greater than 100 tokens, have less than 40% of their characters being alphanumeric or underscores

Natural Language Processing with Transformers

FairScale: A general purpose modular PyTorch library for high performance and large scale training