MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation (2023-10-05T00:00:00.000000Z)

TL;DR

MLAgentBench is introduced, a suite of 13 tasks ranging from improving model performance on CIFAR-10 to recent research problems like BabyLM and identifies several key challenges for LM-based agents such as long-term planning and reducing hallucination.

Abstract

A central aspect of machine learning research is experimentation, the process of designing and running experiments, analyzing the results, and iterating towards some positive outcome (e.g., improving accuracy). Could agents driven by powerful language models perform machine learning experimentation effectively? To answer this question, we introduce MLAgentBench, a suite of 13 tasks ranging from improving model performance on CIFAR-10 to recent research problems like BabyLM. For each task, an agent can perform actions like reading/writing files, executing code, and inspecting outputs. We then construct an agent that can perform ML experimentation based on ReAct framework. We benchmark agents based on Claude v1.0, Claude v2.1, Claude v3 Opus, GPT-4, GPT-4-turbo, Gemini-Pro, and Mixtral and find that a Claude v3 Opus agent is the best in terms of success rate. It can build compelling ML models over many tasks in MLAgentBench with 37.5% average success rate. Our agents also display highly interpretable plans and actions. However, the success rates vary considerably; they span from 100% on well-established older datasets to as low as 0% on recent Kaggle challenges created potentially after the underlying LM was trained. Finally, we identify several key challenges for LM-based agents such as long-term planning and reducing hallucination. Our code is released at https://github.com/snap-stanford/MLAgentBench.

Authors

J. Leskovec

21 papers

Qian Huang

2 papers

Jian Vora

1 papers

TL;DR

Abstract

Authors

References52 items

Mixtral of Experts

Evaluating Language-Model Agents on Realistic Autonomous Tasks

AgentBench: Evaluating LLMs as Agents

WebArena: A Realistic Web Environment for Building Autonomous Agents

Voyager: An Open-Ended Embodied Agent with Large Language Models

SciMON: Scientific Inspiration Machines Optimized for Novelty

AutoML-GPT: Automatic Machine Learning with GPT

Automated Scientific Discovery: From Equation Discovery to Autonomous Discovery Systems

MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks

Generative Agents: Interactive Simulacra of Human Behavior

A Survey on Graph Diffusion Models: Generative AI in Science for Molecule, Protein and Material

Reflexion: language agents with verbal reinforcement learning

GPT-4 Technical Report

House Prices Advanced Regression Techniques

Toolformer: Language Models Can Teach Themselves to Use Tools

Call for Papers - The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus

ReAct: Synergizing Reasoning and Acting in Language Models

The CLRS Algorithmic Reasoning Benchmark

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

WebGPT: Browser-assisted question-answering with human feedback

Nobel Turing Challenge: creating the engine for scientific discovery

Open Graph Benchmark: Datasets for Machine Learning on Graphs

PyTorch: An Imperative Style, High-Performance Deep Learning Library

AutoML: A Survey of the State-of-the-Art

How machine learning won the Higgs boson challenge

Learning Word Vectors for Sentiment Analysis

The Automation of Science

Functional genomic hypothesis generation and experimentation by a robot scientist

Neural architecture search: a survey

JAX: composable transformations of Python+NumPy programs

Learning Multiple Layers of Features from Tiny Images

Fact Check: Confirmed that the data and train.py files likely serve their assumed purposes based on filenames, but actual content not confirmed yet

Research Plan and Status must only include progress that has been made by previous steps. It should not include results not directly confirmed by the previous observation

Follow these instructions and do not forget them

Fact Check: Facts about model architecture and hyperparameters are confirmed by inspecting train

auto-gpt

Understand the train.py script and dataset. Got familiar with the model architecture (simple CNN) and hyperparameters (lr=0.1, momentum=0.9, batch size=128)

Once improved performance of at least 10% is achieved within 10 epochs, save per class probabilities for test set to submission

come up with a high level plan based on your understanding of the problem and available tools and record it in the Research Plan and Status

Highly accurate protein structure

Run experiments with new hyperparameters and model architecture. Done. Tried increasing learning rate to 0.3 and 10 epochs. Achieved test accuracy of 26.35% which improves over baseline by 10%

Come up with a plan to improve the model performance

Research Problem: Given a training script on a dataset train.py, improve upon the current model performance (trained with current hyperparmeters in train.py)

Llama

eor123, GenevievePatterson, and Carlsen,

Do not try installing any new packages or libraries

Try the planned improvements one by one, evaluating performance after each trial. Compare to baseline. Tried increasing LR to 0.3, evaluating performance after running train_lr03

Define a baseline by running the original train.py script. Record accuracy on test set. Baseline test accuracy is 52.53% after 5 epochs

Add regularization (dropout, weight decay, etc) Add dropout with p=0.5 after conv layers and fc layers in train_dropout

Change model architecture (more layers or nodes)

Try different hyperparameters (learning rate, batch size, etc) Increase learning rate to 0.3, decrease batch size to 64 Tried increasing LR to 0

Field of Study

Venue Information

Name

Type

URL

Alternate Names