BENCH 03·LEGAL AI·2025
Putting AI to the Test in Real-World Legal Work
Co-authored study: six AI assistants, two built for lawyers, on 18 extraction questions in-house counsel asked of their own documents. 108 answers, each graded blind by two to four lawyers. No language model graded anything.
/ Overview
A study by Anna Guo and Arthur Souza Rodrigues, published by Legal Benchmarks as Phase 1 in early 2025.
In-house lawyers sent the questions they actually ask of their documents: what does this clause cap, which law governs these eleven contracts, what does the regulation mean by this term. Six assistants answered each one, and lawyers graded every answer without knowing which tool wrote it.
/ What it found
- The most accurate tool was not a legal tool. NotebookLM met the accuracy standard on 14 of 18 tasks. Oliver, ChatGPT and DeepSeek passed 12, GC AI 10, and Copilot 7.
- The two legal tools were the most useful, not the most accurate. GC AI scored 193 of 216 usefulness points and Oliver 171; the best of the rest scored 145.
- Every tool failed where a careful lawyer would not. Asked to define a term the regulation never defines, four of six made one up. Asked for the governing law of eleven contracts, none got all eleven.
- Some failures are invented facts. On a liability clause whose cap is redacted, one tool answered “$1,000,000” and “12 months”. Neither figure is in the contract.
/ Approach
Graded entirely by lawyers, two to four per answer, blind. The study refused LLM grading, citing inconsistent judgments and limited legal expertise, and benchmarks.law keeps that rule. One attempt per task, baseline features, early 2025 versions: the results date quickly, and the method is what lasts.
The copy on benchmarks.law is rewritten in the form of the other benchmarks.law reports, with the charts drawn from the data behind the source page.