DOBF: A Deobfuscation Pre-Training Objective for Programming Languages
Published in
Neural Information Processing Systems(2021)External Links:
TL
TL;DR
A new pre-training objective, DOBF, is introduced that leverages the structural aspect of programming languages and pre-trains a model to recover the original version of obfuscated source code and shows that models pre-trained with DOBF significantly outperform existing approaches on multiple downstream tasks.
Abstract
Recent advances in self-supervised learning have dramatically improved the state of the art on a wide variety of tasks. However, research in language model pre-training has mostly focused on natural languages, and it is unclear whether models like BERT and its variants provide the best pre-training when applied to other modalities, such as source code. In this paper, we introduce a new pre-training objective, DOBF, that leverages the structural aspect of programming languages and pre-trains a model to recover the original version of obfuscated source code. We show that models pre-trained with DOBF significantly outperform existing approaches on multiple downstream tasks, providing relative improvements of up to 13% in unsupervised code translation, and 24% in natural language code search. Incidentally, we found that our pre-trained model is able to de-obfuscate fully obfuscated source files, and to suggest descriptive variable names.
References62 items
1
Exploiting Method Names to Improve Code Summarization: A Deliberation Multi-Task Learning Approach
2
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
3
Industry-Scale IR-Based Bug Localization: A Perspective from Facebook
4
Neural reverse engineering of stripped binaries using augmented control flow graphs
5