Assemble Foundation Models for Automatic Code Summarization

Published in

IEEE International Conference on Software Analy...(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

This work assembles available foundation models, such as CodeBERT and GPT-2, into a single model named AdaMo, and utilizes Gaussian noise as the simulation of contextual information to optimize the latent representation.

Abstract

Automatic code summarization is beneficial to software development and maintenance since it reduces the burden of manual tasks. Currently, artificial intelligence is undergoing a paradigm shift. The foundation models pretrained on massive data and finetuned to downstream tasks surpass specially customized models. This trend inspired us to consider reusing foundation models instead of learning from scratch. Based on this, we propose a flexible and robust approach for automatic code summarization based on neural networks. We assemble available foundation models, such as CodeBERT and GPT-2, into a single model named AdaMo. Moreover, we utilize Gaussian noise as the simulation of contextual information to optimize the latent representation. Furthermore, we introduce two adaptive schemes from the perspective of knowledge transfer, namely continuous pretraining and intermediate finetuning, and design intermediate stage tasks for general sequence-to-sequence learning. Finally, we evaluate AdaMo against a benchmark dataset for code summarization, by comparing it with state-of-the-art models.

Authors

Jian Gu

1 papers

P. Salza

1 papers

H. Gall

1 papers

References76 items

Reassessing automatic evaluation metrics for code summarization tasks

On the Opportunities and Risks of Foundation Models

Evaluating Large Language Models Trained on Code

Multimodal Representation for Neural Code Search

Pre-Trained Models: Past, Present and Future

Assemble Foundation Models for Automatic Code Summarization

Published in

IEEE International Conference on Software Analy...(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Jian Gu

1 papers

P. Salza

1 papers

H. Gall

1 papers

References76 items

Reassessing automatic evaluation metrics for code summarization tasks

On the Opportunities and Risks of Foundation Models

Evaluating Large Language Models Trained on Code

Multimodal Representation for Neural Code Search

Pre-Trained Models: Past, Present and Future

Improving Code Summarization with Block-wise Abstract Syntax Tree Splitting

Unified Pre-training for Program Understanding and Generation

CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

Muppet: Massive Multi-task Representations with Pre-Finetuning

Code Summarization with Structure-induced Transformer

On Losses for Modern Language Models

A Decade Survey of Transfer Learning (2010–2020)

Transformers: State-of-the-Art Natural Language Processing

Code to Comment “Translation”: Data, Metrics, Baselining & Evaluation

SummEval: Re-evaluating Summarization Evaluation

Contrastive Code Representation Learning

Self-Attention Guided Copy Mechanism for Abstractive Summarization

Retrieval-based Neural Source Code Summarization

Retrieval-Augmented Generation for Code Summarization via Hybrid GNN

Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation

Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?

A Transformer-based Approach for Source Code Summarization

Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

FELIX: Flexible Text Editing Through Tagging and Insertion

CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Deep Learning for Source Code Modeling and Generation

PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

Automatic Code Summarization: A Systematic Literature Review

Leveraging Pre-trained Checkpoints for Sequence Generation Tasks

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Learning Blended, Precise Semantic Program Embeddings

Deep code comment generation with hybrid lexical and syntactical information

Learning Scalable and Precise Representation of Program Semantics

MASS: Masked Sequence to Sequence Pre-training for Language Generation

A Novel Neural Source Code Representation Based on Abstract Syntax Tree

Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches

A Neural Model for Generating Natural Language Summaries of Program Subroutines

Self-organizing maps for storage and transfer of knowledge in reinforcement learning

Improving Automatic Source Code Summarization via Deep Reinforcement Learning

Bottom-Up Abstractive Summarization

Deep Code Comment Generation

Sequential Copying Networks

A general path-based representation for predicting program properties

Self-Attention with Relative Position Representations

Learning to Represent Programs with Graphs

Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation

Attention is All you Need

Get To The Point: Summarization with Pointer-Generator Networks

Zero-Shot Learning — The Good, the Bad and the Ugly

Summarizing Source Code using a Neural Attention Model

Modeling Coverage for Neural Machine Translation

Gated Graph Sequence Neural Networks

A Neural Attention Model for Abstractive Sentence Summarization

Effective Approaches to Attention-based Neural Machine Translation

Sequence to Sequence Learning with Neural Networks

Neural Machine Translation by Jointly Learning to Align and Translate

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

A Survey on Transfer Learning

METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments

Dimensionality Reduction by Learning an Invariant Mapping

ROUGE: A Package for Automatic Evaluation of Summaries

Bleu: a Method for Automatic Evaluation of Machine Translation

Long Short-Term Memory

Recent Advances in Language Model Fine-Tuning

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Improving Language Understanding by Generative Pre-Training

RQ1 How well does our transfer learning model perform in code summarization?

RQ3 How well does code summarization beneﬁt from adaptive continuous pretraining?

Field of Study

Computer Science

Journal Information

Name

2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)

Page

935-946

Venue Information

Name

IEEE International Conference on Software Analysis, Evolution, and Reengineering

Type

conference

URL

http://www.informatik.uni-stuttgart.de/ifi/ps/reengineering/

Alternate Names

IEEE Int Conf Softw Anal Evol Reengineering
SANER

TL;DR

Abstract

Authors

References76 items

Reassessing automatic evaluation metrics for code summarization tasks

On the Opportunities and Risks of Foundation Models

Evaluating Large Language Models Trained on Code

Multimodal Representation for Neural Code Search

Pre-Trained Models: Past, Present and Future

TL;DR

Abstract

Authors

References76 items

Reassessing automatic evaluation metrics for code summarization tasks

On the Opportunities and Risks of Foundation Models

Evaluating Large Language Models Trained on Code

Multimodal Representation for Neural Code Search

Pre-Trained Models: Past, Present and Future

Improving Code Summarization with Block-wise Abstract Syntax Tree Splitting

Unified Pre-training for Program Understanding and Generation

CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

Muppet: Massive Multi-task Representations with Pre-Finetuning

Code Summarization with Structure-induced Transformer

On Losses for Modern Language Models

A Decade Survey of Transfer Learning (2010–2020)

Transformers: State-of-the-Art Natural Language Processing

Code to Comment “Translation”: Data, Metrics, Baselining & Evaluation

SummEval: Re-evaluating Summarization Evaluation

Contrastive Code Representation Learning

Self-Attention Guided Copy Mechanism for Abstractive Summarization

Retrieval-based Neural Source Code Summarization

Retrieval-Augmented Generation for Code Summarization via Hybrid GNN

Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation

Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?

A Transformer-based Approach for Source Code Summarization

Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

FELIX: Flexible Text Editing Through Tagging and Insertion

CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Deep Learning for Source Code Modeling and Generation

PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

Automatic Code Summarization: A Systematic Literature Review

Leveraging Pre-trained Checkpoints for Sequence Generation Tasks

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Learning Blended, Precise Semantic Program Embeddings

Deep code comment generation with hybrid lexical and syntactical information

Learning Scalable and Precise Representation of Program Semantics

MASS: Masked Sequence to Sequence Pre-training for Language Generation

A Novel Neural Source Code Representation Based on Abstract Syntax Tree

Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches

A Neural Model for Generating Natural Language Summaries of Program Subroutines

Self-organizing maps for storage and transfer of knowledge in reinforcement learning

Improving Automatic Source Code Summarization via Deep Reinforcement Learning

Bottom-Up Abstractive Summarization

Deep Code Comment Generation

Sequential Copying Networks

A general path-based representation for predicting program properties

Self-Attention with Relative Position Representations

Learning to Represent Programs with Graphs

Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation

Attention is All you Need

Get To The Point: Summarization with Pointer-Generator Networks

Zero-Shot Learning — The Good, the Bad and the Ugly

Summarizing Source Code using a Neural Attention Model

Modeling Coverage for Neural Machine Translation

Gated Graph Sequence Neural Networks

A Neural Attention Model for Abstractive Sentence Summarization

Effective Approaches to Attention-based Neural Machine Translation

Pointer Networks

Answering

Sequence to Sequence Learning with Neural Networks

Neural Machine Translation by Jointly Learning to Align and Translate

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

A Survey on Transfer Learning

METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments

Dimensionality Reduction by Learning an Invariant Mapping