To enable cross-disciplinary conversations about LLMs in the law, it is shown how popular legal frameworks for describing legal reasoning correspond to LegalBench tasks, thus giving lawyers and LLM developers a common vocabulary.
The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LegalBench: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LegalBench was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning -- which distinguish between its many forms -- correspond to LegalBench tasks, thus giving lawyers and LLM developers a common vocabulary. This paper describes LegalBench, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LegalBench enables.
Christopher Ré
6 papers
Neel Guha
7 papers
Daniel E. Ho
3 papers
Julian Nyarko
3 papers
D. Rockmore
1 papers
Nils Holzenberger
3 papers
Faiz Surani
2 papers
Joel Niklaus
4 papers
Peter Henderson
2 papers
Adam Chilton
1 papers
Aditya Narayana
1 papers
Alex Chohlas-Wood
1 papers
Austin M. K. Peters
1 papers
Brandon Waldon
1 papers
Diego A. Zambrano
1 papers
Dmitry Talisman
1 papers
E. Hoque
1 papers
F. Fagan
1 papers
Galit Sarfaty
1 papers
Gregory M. Dickinson
1 papers
Haggai Porat
1 papers
Jason Hegland
1 papers
Jessica Wu
1 papers
Joe Nudell
1 papers
John J. Nay
1 papers
Jonathan H. Choi
1 papers
K. Tobia
1 papers
M. Hagan
1 papers
Megan Ma
1 papers
Michael A. Livermore
1 papers
Nikon Rasumov-Rahe
1 papers
Noam Kolt
1 papers
Sean Rehaag
1 papers
Sharad Goel
1 papers
Shangsheng Gao
1 papers
Spencer Williams
1 papers
S. Gandhi
1 papers
Tomer Zur
1 papers
Varun J. Iyer
1 papers
Zehua Li
1 papers