CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

Published in

arXiv.org(2019)

External Links:

Generate Graph

TL;DR

The methodology used to obtain the corpus and expert labels, as well as a number of simple baseline solutions for the task are described.

Abstract

Semantic code search is the task of retrieving relevant code given a natural language query. While related to other information retrieval tasks, it requires bridging the gap between the language used in code (often abbreviated and highly technical) and natural language more suitable to describe vague concepts and ideas. To enable evaluation of progress on code search, we are releasing the CodeSearchNet Corpus and are presenting the CodeSearchNet Challenge, which consists of 99 natural language queries with about 4k expert relevance annotations of likely results from CodeSearchNet Corpus. The corpus contains about 6 million functions from open-source code spanning six programming languages (Go, Java, JavaScript, PHP, Python, and Ruby). The CodeSearchNet Corpus also contains automatically generated query-like natural language for 2 million functions, obtained from mechanically scraping and preprocessing associated function documentation. In this article, we describe the methodology used to obtain the corpus and expert labels, as well as a number of simple baseline solutions for the task. We hope that CodeSearchNet Challenge encourages researchers and practitioners to study this interesting task further and will host a competition and leaderboard to track the progress on the challenge. We are also keen on extending CodeSearchNet Challenge to more queries and programming languages in the future.

Authors

Marc Brockschmidt

4 papers

Miltiadis Allamanis

5 papers

Hamel Husain

1 papers

References32 items

When deep learning met code search

CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning

An Introduction to Neural Information Retrieval

The adverse effects of code duplication in machine learning models of code

A Retrieve-and-Edit Framework for Predicting Structured Outputs

CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

Published in

arXiv.org(2019)

External Links:

Generate Graph

TL;DR

The methodology used to obtain the corpus and expert labels, as well as a number of simple baseline solutions for the task are described.

Abstract

Authors

Marc Brockschmidt

4 papers

Miltiadis Allamanis

5 papers

Hamel Husain

1 papers

References32 items

When deep learning met code search

CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning

An Introduction to Neural Information Retrieval

The adverse effects of code duplication in machine learning models of code

A Retrieve-and-Edit Framework for Predicting Structured Outputs

Tiferet Gazit

1 papers

Structured Neural Summarization

Mapping Language to Code in Programmatic Context

code2seq: Generating Sequences from Structured Representations of Code

StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow

NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System

DéjàVu: a map of code duplicates on GitHub

A Survey of Machine Learning for Big Code and Naturalness

A Parallel Corpus of Python Functions and Documentation Strings for Automated Code Documentation and Code Generation

Attention is All you Need

Deep API learning

Latent Predictor Networks for Code Generation

A Convolutional Attention Network for Extreme Summarization of Source Code

Neural Machine Translation of Rare Words with Subword Units

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

Convolutional Neural Networks for Sentence Classification

The Linux Operating System

A new algorithm for data compression

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

A Syntactic Neural Model for General-Purpose Code Generation

An Introduction to Information Retrieval

Context Some results were semantically correct, but not relying on related helper functions and thus not self-contained

Bidirectional RNN models where we employ the GRU cell [7] to summarize the input sequence

Neural Bag of Words where each (sub)token is embedded to a learnable embedding (vector representation)

Code Quality A subset of the results being returned are functionally correct code, but of low quality, even though they originated in reasonably popular projects

Library vs. Project Specific Often a search yields code that is very specific to a given project ( e.g. using internal utility functions), whereas other times the code is very general and verbose

Field of Study

Computer ScienceMathematics

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

arXiv.org

Type

URL

https://arxiv.org

Alternate Names

ArXiv

TL;DR

Abstract

Authors

References32 items

When deep learning met code search

CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning

An Introduction to Neural Information Retrieval

The adverse effects of code duplication in machine learning models of code

A Retrieve-and-Edit Framework for Predicting Structured Outputs

TL;DR

Abstract

Authors

References32 items

When deep learning met code search

CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning

An Introduction to Neural Information Retrieval

The adverse effects of code duplication in machine learning models of code

A Retrieve-and-Edit Framework for Predicting Structured Outputs

Structured Neural Summarization

Mapping Language to Code in Programmatic Context

code2seq: Generating Sequences from Structured Representations of Code

Deep Code Search

StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow

NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System

DéjàVu: a map of code duplicates on GitHub

A Survey of Machine Learning for Big Code and Naturalness

A Parallel Corpus of Python Functions and Documentation Strings for Automated Code Documentation and Code Generation

Attention is All you Need

Deep API learning

Latent Predictor Networks for Code Generation

A Convolutional Attention Network for Extreme Summarization of Source Code

Neural Machine Translation of Rare Words with Subword Units

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

Convolutional Neural Networks for Sentence Classification

The Linux Operating System

A new algorithm for data compression

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

A Syntactic Neural Model for General-Purpose Code Generation

An Introduction to Information Retrieval

Context Some results were semantically correct, but not relying on related helper functions and thus not self-contained

Bidirectional RNN models where we employ the GRU cell [7] to summarize the input sequence

Neural Bag of Words where each (sub)token is embedded to a learnable embedding (vector representation)

Query Ambiguity

Code Quality A subset of the results being returned are functionally correct code, but of low quality, even though they originated in reasonably popular projects

Library vs. Project Specific Often a search yields code that is very specific to a given project ( e.g. using internal utility functions), whereas other times the code is very general and verbose

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names