Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation (2023-06-09T00:00:00.000000Z)

TL;DR

Evaluation of the 47 cutting-edge LLMs on Xiezhi indicates that LLMs exceed average performance of humans in science, engineering, agronomy, medicine, and art, but fall short in economics, jurisprudence, pedagogy, literature, history, and management.

Abstract

New Natural Langauge Process~(NLP) benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present Xiezhi, the most comprehensive evaluation suite designed to assess holistic domain knowledge.Xiezhi comprises multiple-choice questions across 516 diverse disciplines ranging from 13 different subjects with 249,587 questions and accompanied by Xiezhi-Specialty with 14,041 questions and Xiezhi-Interdiscipline with 10,746 questions. We conduct evaluation of the 47 cutting-edge LLMs on Xiezhi. Results indicate that LLMs exceed average performance of humans in science, engineering, agronomy, medicine, and art, but fall short in economics, jurisprudence, pedagogy, literature, history, and management. All the evaluation code and data are open sourced in https://github.com/MikeGu721/XiezhiBenchmark

Authors

Zihan Li

2 papers

Yanghua Xiao

9 papers

Qi He

2 papers

TL;DR

Abstract

Authors

References85 items

CMMLU: Measuring massive multitask language understanding in Chinese

Holistic Evaluation of Language Models

M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Models

C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models

Measuring Massive Multitask Chinese Understanding

Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluation

OpenAssistant Conversations - Democratizing Large Language Model Alignment

HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge

AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models

DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

A Survey of Large Language Models

Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases

Sparks of Artificial General Intelligence: Early experiments with GPT-4

GPT-4 Technical Report

Is ChatGPT a Good NLG Evaluator? A Preliminary Study

LLaMA: Open and Efficient Foundation Language Models

A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT

Specializing Smaller Language Models towards Multi-Step Reasoning

LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain

Discovering Language Model Behaviors with Model-Written Evaluations

Constitutional AI: Harmlessness from AI Feedback

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

Scaling Instruction-Finetuned Language Models

Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

GLM-130B: An Open Bilingual Pre-trained Model

Emergent Abilities of Large Language Models

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

GPT-NeoX-20B: An Open-Source Autoregressive Language Model

Training language models to follow instructions with human feedback

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning

LoRA: Low-Rank Adaptation of Large Language Models

Measuring Coding Challenge Competence With APPS

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

Neural Correction Model for Open-Domain Named Entity Recognition.

A Survey of Evaluation Metrics Used for NLG Systems

Language Models are Few-Shot Learners

PIQA: Reasoning about Physical Commonsense in Natural Language

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning

HellaSwag: Can a Machine Really Finish Your Sentence?

Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

Free dolly: Introducing the world’s first truly open instruction-tuned llm

ChatDoctor: A Medical Chat Model Fine-tuned on LLaMA Model using Medical Domain Knowledge

Crosslingual Generalization through Multitask Finetuning

Chatgpt: Optimizing language models for dialogue

Luotuo: An instruction-following chinese language model, lora tuning on llama

Koala: A dialogue model for academic research

Introducing mpt-7b: A new standard for open-source, ly usable llms

2023a). Fudannlp moss

h2ogpt -the world's best open source gpt

Chatglm-med: 基于中文医学知识 的chatglm模型微调

Baize: An open-source chat model with parameter-efficient tuning on self-chat data

Stanford alpaca: An instruction-following llama model

How does gpt obtain its ability? tracing emergent abilities of language models to their sources

An empirical analysis of compute-optimal large language model training

Language Models are Unsupervised Multitask Learners

If you are using existing assets (e.g., code, data, models) or curating/releasing new assets.

If deletion of a question leads to imbalance, new questions should be re-selected from Xiezhi-All to be included

ControlScienceandEngineering ControlTheoryandControlEngineering TestingTechnologyandAutomation SystemsEngineering PatternRecognitionandIntelligentSystems Navigation,GuidanceandControl

Most benchmarks relied on 4-option multiple choice questions. This made it too easy for models to guess correctly. More options were needed to better differentiate model capabilities

Did you mention the license of the assets?

Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Provided in

(a) If your work uses existing assets, did you cite the creators? [Yes] We use baseline models from Huggingface’s Transformers, and detail describe and cite them in Appendix Models

c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We provide details in Appendix Models

a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] The instructions are included in

Did you discuss whether and how consent was obtained from people whose data you ’ re using/curating?

Questions involving sensitive content such as military matters and politics should be removed

If the subject annotation is incorrect, it should be removed

If issue that cannot be modified was introduced during the crawling of questions, it should be removed

Many existing benchmarks quickly became outdated as they got incorporated into the training data of the latest LLMs. There was a need for benchmarks with fresher data

StabilityAI (2023)

Have you read the ethics review guidelines and ensured that your paper conforms to them?

If you are including theoretical results

Chatglm-med: 基于中文医学知识的chatglm模型微调