Learning and Evaluating Contextual Embedding of Source Code

This paper curates a massive, deduplicated corpus of 7.4M Python files from GitHub, and creates an open-sourced benchmark that comprises five classification tasks and one program-repair task, akin to code-understanding tasks proposed in the literature before, showing that CuBERT outperforms them all, even with shorter training, and with fewer labeled examples.