ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation (2023-08-03T00:00:00.000000Z)

TL;DR

This work makes the first attempt to evaluate LLMs in a more challenging code generation scenario, i.e. class-level code generation, and finds the limited model ability of generating method-dependent code and discusses the frequent error types in generated classes.

Abstract

In this work, we make the first attempt to evaluate LLMs in a more challenging code generation scenario, i.e. class-level code generation. We first manually construct the first class-level code generation benchmark ClassEval of 100 class-level Python code generation tasks with approximately 500 person-hours. Based on it, we then perform the first study of 11 state-of-the-art LLMs on class-level code generation. Based on our results, we have the following main findings. First, we find that all existing LLMs show much worse performance on class-level code generation compared to on standalone method-level code generation benchmarks like HumanEval; and the method-level coding ability cannot equivalently reflect the class-level coding ability among LLMs. Second, we find that GPT-4 and GPT-3.5 still exhibit dominate superior than other LLMs on class-level code generation, and the second-tier models includes Instruct-Starcoder, Instruct-Codegen, and Wizardcoder with very similar performance. Third, we find that generating the entire class all at once (i.e. holistic generation strategy) is the best generation strategy only for GPT-4 and GPT-3.5, while method-by-method generation (i.e. incremental and compositional) is better strategies for the other models with limited ability of understanding long instructions and utilizing the middle information. Lastly, we find the limited model ability of generating method-dependent code and discuss the frequent error types in generated classes. Our benchmark is available at https://github.com/FudanSELab/ClassEval.

Authors

References50 items

PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback

Can Large Language Models Write Good Property-Based Tests?

A Survey on Evaluation of Large Language Models

Lost in the Middle: How Language Models Use Long Contexts

Recommender Systems in the Era of Large Language Models (LLMs)

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

Judging LLM-as-a-judge with MT-Bench and Chatbot Arena

Benchmarking Foundation Models with Language-Model-as-an-Examiner

StarCoder: may the source be with you!

Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation

The promise and peril of using a large language model to obtain clinical information: ChatGPT performs strongly as a fertility counseling tool with limitations.

Explainable automated debugging via large language model-driven scientific debugging

CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X

ChatGPT Is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models

GPT-4 Technical Report

Language Is Not All You Need: Aligning Perception with Language Models

LLaMA: Open and Efficient Foundation Language Models

A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity

Learning-based Widget Matching for Migrating GUI Test Cases

SantaCoder: don't reach for the stars!

Large Language Models Meet NL2Code: A Survey

DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation

Multi-lingual Evaluation of Code Generation Models

GLM-130B: An Open Bilingual Pre-trained Model

Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction

PanGu-Coder: Program Synthesis with Function-Level Language Modeling

InCoder: A Generative Model for Code Infilling and Synthesis

A systematic evaluation of large language models of code

Competition-level code generation with AlphaCode

Program Synthesis with Large Language Models

Evaluating Large Language Models Trained on Code

Measuring Coding Challenge Competence With APPS

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

The Curious Case of Neural Text Degeneration

Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow

Discrete Signal Processing on Graphs: Sampling Theory

Evaluating the efficacy of test-driven development: industrial case studies

Applying 'design by contract'

Enabling Programming Thinking in Large Language Models Toward Code Generation

Instruct-codegen

A Conversational Paradigm for Program Synthesis

Dense-6.7b

Language Models are Unsupervised Multitask Learners

Python – The Fastest Growing Programming Language

Classeval github

Coverage library

Code generation datasets in huggingface

Unittest framework

Feng Chaofeng Sha

Openai api interface