FrontierCode: How To Tell Whether A Maintainer Would Merge Your AI’s Pull Request

🔍 Read the full analysis: FrontierCode: How To Tell Whether A Maintainer Would Merge Your AI’s Pull Request on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get movie nights delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Artificial Analysis evaluated 23 AI models on FrontierCode, a coding benchmark that scores patches against maintainer-style review criteria rather than relying on test results alone. Claude Opus 5.5 ranked first with a 52.2% score, but cleared every blocking check in 57.7% of attempts, leaving a substantial share of attempts short of the benchmark’s merge threshold.

The results, added to Artificial Analysis’s evaluations on October 12, 2026, place GPT-6.1 Sol second at 49.1% and Claude Fable 5.1 third at 49.0%. Artificial Analysis reports a separate pass rate: Opus cleared every blocking check in 57.7% of attempts, while GPT-6.1 Sol did so in 54.7%. The score and pass rate are distinct measures; the score also reflects quality checks.

FrontierCode’s full benchmark contains 150 tasks, while Artificial Analysis ran its 100-task Main set. Tasks come from established repositories and were written by maintainers, according to the source report. The evaluation used the same open-source agent harness, Stirrup, for each model, with up to 250 turns per attempt and three attempts per task. Nine tasks were not solved by any tested model on any attempt.

A model receives zero for an attempt if it fails a blocking check, even when other checks pass. The benchmark also penalizes issues such as oversized diffs, edits outside the requested scope, and tests that do not establish the proposed change’s correctness. Artificial Analysis says the code-review step used GPT-6 Luna (medium) as the judge. These design choices make the results a measure of performance under this benchmark’s rules, not a guarantee of how a particular maintainer would judge a real pull request.

The Cost of a Merge-Ready Patch

The results challenge the idea that passing automated tests alone is enough to hand a repository change to an AI agent. FrontierCode checks for scope, regression risks and review quality as well as correctness. On this evaluation, even the highest-ranked model failed to clear at least one blocking check in more than four in ten attempts. That makes human review relevant to any team considering how much responsibility to give coding agents.

Cost varied sharply in Artificial Analysis’s figures: GPT-6 Luna (max) scored 41.9% at $0.09 per task, while Claude Opus 5.5 scored 52.2% at $10.15. GPT-6.1 Sol scored 49.1% at $0.81. The source characterizes Opus’s three-point advantage over Sol as costing 12.5 times as much. Those figures may help teams weigh quality against usage cost, but they are benchmark-specific and do not establish the cost or success rate for every codebase or deployment.

The ranking also differs from general intelligence scores in the source’s comparison. Claude Sonnet 5.5 placed second among the listed models on the Intelligence Index but fifth on FrontierCode; Grok 4.7 ranked 18th on FrontierCode in the report. This suggests general model rankings alone may not predict performance on this particular style of repository work.

How FrontierCode Scores Repository Work

FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026, according to the source report. Version 1.1 followed in July with rules intended to prevent agents from looking up solutions online. More than 20 maintainers wrote tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost. The report says each task took more than 40 hours to prepare.

The benchmark checks patches across several dimensions: hidden tests, command checks, tests that must fail on the original code, adaptive tests for complex behavior, limits on files and diff size, and an LLM review against maintainers’ criteria. Some checks are blocking; others affect the score after the blocking requirements are met. An attempt flagged by Cognition’s trajectory scanner for looking up a solution also receives zero, according to the source.

Artificial Analysis’s results cover its 100-task Main set, not all 150 FrontierCode tasks. Its process caps each attempt at 250 turns and permits three attempts per task. The source quotes Celery maintainer Tomer Nosrati describing the benchmark as grading “like a tech lead,” in contrast with evaluations that focus on continuous-integration results.

“Where others grade like a CI, FrontierCode grades like a tech lead.”

— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement

Limits of the Published Results

The results do not establish how often these models would produce acceptable changes across all repositories, task types or review teams. FrontierCode uses a private task set and a defined harness, turn limit and judging process; real-world outcomes may differ. The source also does not provide a result for every model discussed in the broader market: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of October 13, 2026.

It is also unclear from the supplied report how the scores would change across the full 150-task benchmark, or how well the LLM review judge matches decisions made by human maintainers. The reported costs and rankings should be read as results of this particular evaluation, rather than direct forecasts for a team’s own repository or production workload.

Further Model Results to Watch

Artificial Analysis may add results as further models are evaluated; the source identifies several prominent models without scores as of October 13, including Muse Wonder Full. Their eventual FrontierCode performance could change the ranking and make comparisons with other coding evaluations possible.

For now, the published figures give developers a basis for comparing models on a task set designed around repository review, alongside the reported price per task. Teams applying the results will still need to test agents against their own code, rules and review practices. The next useful milestone is broader, comparable evaluation data—not a conclusion that any model can reliably submit changes without human oversight.

Key Questions

What does FrontierCode measure?

It evaluates whether an AI-generated code patch meets a range of repository and review checks, including hidden tests, scope limits and code-review criteria. Passing all blocking checks is required to avoid a zero for an attempt.

Which model ranked first?

Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s results, with a 52.2% score and a 57.7% rate of clearing all blocking checks.

Does the top result mean AI can merge code without review?

No. The result means Opus cleared all blocking checks in 57.7% of attempts under this benchmark’s setup. It does not establish that AI-generated changes are safe to merge without human review in other repositories.

How much did the models cost per task?

Reported costs ranged from $0.09 for GPT-6 Luna (max) to $10.15 for Claude Opus 5.5 among the cost figures given. These are Artificial Analysis benchmark costs and may not match other workloads.

Which models were missing from the results?

As of October 13, 2026, the source report listed Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no FrontierCode result.

Source: ThorstenMeyerAI.com