🔍 Read the full analysis: How Z.ai’s Legal AI Performs On Benchmark Tasks—and Why It Ranks Near The Top on ThorstenMeyerAI.com
Get movie nights delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh among 81 models in the archived results. It passed 92.5% of individual criteria, but a task counted as complete only when every criterion passed. The result is a benchmark measure, not evidence that its legal output can be used without professional review.
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh of 81 models on Vals AI’s archived Harvey Legal Agent Benchmark leaderboard, passing every required criterion in 19.17% of tested tasks. The score, which Z.ai rounds to 19.2%, reflects a strict all-or-nothing grading rule: a task fails if even one rubric requirement is missed, so it does not mean the model got only one-fifth of the individual criteria right.
Vals AI’s archived Version 1 leaderboard lists the legal model at 19.17% of tasks and $0.15 per test. The same leaderboard gives it a 92.5% pass rate on individual rubric criteria. Those figures describe different measures: the first counts only tasks where every criterion passed, while the second counts individual pass/fail requirements. Z.ai’s documentation reports the rounded task figure as 19.2% and says the model improved by 12.5 percentage points over the base model.
The comparison with the base model is supported by the same Vals table: GLM-5.3-Flash scored 6.67% on the all-criteria task measure, 12.50 percentage points below GLM Legal. That is a difference in percentage points, not a claim that the legal model is 12.5 times better. The archived leaderboard lists other systems above it, including Muse Spark 1.2 at 25.42%, Muse Spark 1.3 Max at 23.75%, and Gemini 4 Argon at 19.58%.
Z.ai describes GLM-5.3-FlashX-Legal as a model for legal research, document understanding, contract review and drafting, and task execution. Its documentation says it underwent legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of task environments modeled on legal workflows. Z.ai recommends connecting the model to external services for laws, cases and company information. Its pricing page lists $0.37 per million input tokens and $1.25 per million output tokens; that pricing is separate from Vals’s reported $0.15 per benchmark test.
What the Score Means for Legal Work
The result points to a potential cost-performance tradeoff for firms and legal teams evaluating AI agents. In Vals’s archived results, GLM Legal’s reported $0.15 per test is lower than the listed costs for higher-ranked models, while its all-criteria result places it near the top of the 81-model table. A low test cost and a high ranking do not, on their own, establish that a system is suitable for a particular client matter or workflow.
The gap between 92.5% of criteria passed and 19.17% of tasks fully passed also illustrates why benchmark scores need context. A task may contain many separate requirements, and missing even one can turn a mostly correct work product into a failed task under this grading approach. For legal users, the score highlights the need to inspect the underlying work, check citations and facts, and decide how errors affect the intended use.
The source report describes a four-task spot check in which the legal model cited rulings the author could not find. That small check is not a benchmark result and cannot establish a general error rate. It does, however, underline a practical limitation: leaderboard performance cannot replace case-by-case verification, particularly where a citation or deadline could affect legal advice or a filing.
Popular right now
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
KitchenAid Artisan 5-Quart Tilt-Head Stand Mixer, Almond Cream
Dyson V8 Cyclone Cordless Vacuum
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Agents
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. The benchmark contains 1,250 tasks across 24 legal practice areas. Each task provides instructions and matter materials and asks an agent to produce work for review. Harvey says the tasks are evaluated against more than 75,000 expert-written rubric criteria, covering items such as facts, conclusions, citations, recommendations, deadlines and formatting.
Under Harvey’s all-pass grading rule, every criterion must pass for the task to count as complete. Vals AI’s leaderboard uses a private holdout set that it says contains 120 tasks and mirrors the public task mix; internet access is disabled during testing. According to Vals, two AI judges grade each submission and the reported task score averages their pass rates. This is the basis for the headline percentage, rather than a direct review of every possible legal use.
Z.ai’s AutoClaw blog announced the model on October 13, 2026, with a beta available by application. Vals lists the release date as October 12. Z.ai says the system is built on GLM-5.3-Flash. Its documentation calls the model optimized for professional legal workflows, but the documentation page cited in the source report does not carry a publication date.
“An all-criteria pass rate of 19.2%”
— Z.ai, in its model documentation
Limits of the Published Results
Z.ai calls the result state-of-the-art performance among open-source models, but the source material does not identify the comparison set behind that claim, and the report says it found no published weights for the legal model. The available leaderboard supports the model’s seventh-place overall position in the archived Version 1 results; it does not by itself settle how the model compares across every open-source system or evaluation.
The score applies to a 120-task private holdout graded using two AI judges, not to all legal work or every way a law firm might configure the model. The source report also notes that its own four-task spot check found rulings it could not locate in the legal model’s citations. Four tasks are too few to estimate error rates, and the cited material does not provide a larger independent audit of citation accuracy, performance across practice areas, or results under different deployment conditions.
Other practical details remain unreported in the source material, including which parties process client data, the applicable data-handling rules, and what review controls Z.ai or deploying firms recommend before output is used. The benchmark measures task completion under its rubric; it does not establish that the model provides legal advice, meets a firm’s confidentiality obligations, or is safe for unsupervised use.
What Buyers Should Verify
Legal teams considering the model will need to test it on representative work and set review requirements suited to the consequences of errors. That includes checking authorities and citations against primary sources, verifying factual statements and deadlines, and defining who approves drafts before they leave the organization. Those steps are operational safeguards, not findings established by the leaderboard.
Further evidence would help clarify the model’s position: published weights or a defined comparison set for Z.ai’s open-source claim; results on additional independent evaluations; and more detail about data handling, access controls and the conditions under which client materials are processed. Z.ai’s announcement opened a beta by application, but the provided source material does not give a broader release schedule or announce a next benchmark update.
Key Questions
What does the 19.17% score measure?
It is GLM-5.3-FlashX-Legal’s all-criteria task pass rate on Vals AI’s archived Harvey benchmark leaderboard. A task counts as passed only when every required rubric criterion passes.Why is the individual-criteria score much higher?
Vals reports a 92.5% criteria pass rate for GLM Legal, but a task can contain many separate requirements. Passing most criteria is not enough for an all-pass task score if even one criterion fails.How did the model rank?
It ranked seventh of 81 models in Vals AI’s archived Version 1 leaderboard. That placement applies to the listed benchmark results, not to every legal AI product or legal task.Does the benchmark show that the model is safe to use without review?
No. The benchmark measures performance on a defined set of tasks and criteria. It does not establish that citations, advice or other output can be used without professional verification.Is Z.ai’s open-source performance claim independently settled?
The source report says Z.ai does not name the comparison set for its claim of state-of-the-art performance among open-source models, and it found no published weights for the legal model. The archived leaderboard supports the model’s overall rank, but does not resolve those separate questions.Source: ThorstenMeyerAI.com
