By Chris Hinckley, Andrea Tan, Sherry Han, Estelle Hu, Meiru Han, Zianyi Wang, Om Babar, Erin Lafrenz and Aaliyah Brown
Over the past year the choices for running a coding agent against our codebase have multiplied: more models than any one engineer can keep track of, several harnesses to drive them, a reasoning-effort setting on most of them, and a growing set of tools that promise to cut the token bill. Each vendor publishes benchmark numbers. None of them were produced on a 6-million-line Java platform with Gradle test cycles of five to ten minutes, so we had no way to know which of those numbers applied to us.
We built an internal benchmark, Touchstone, to answer that for ourselves. It mines tasks from real pull requests merged into our repositories, runs any model under any harness in a sealed copy of the repository, and grades each attempt by running the tests the original engineers wrote. No model is asked to judge another model's work. The results below come from about 34,000 graded attempts run between September and October 2026. This post covers what we found across three questions: which model and harness, whether the repository's own agent instructions help, and whether context-compression tools save money.
What we found
-
The top of the table is a statistical tie. Seven combinations landed within 2.7 points of each other, led by SWE-2 under Devin CLI at 71.7% and Claude Opus 5 under Claude Code at 70.8%. With 200 tasks and three attempts each, a lead has to reach about 6.6 points before it is distinguishable from noise for one pair, and 11.5 points once you correct for comparing every pair against every other.
-
Cost per attempt among those seven spans a factor of nearly forty. GPT-6.1 Sol under Claude Code solved 69.7% of tasks at $0.11 per attempt. Claude Opus 5 under OpenCode solved 69.2% at $4.21. The cost-quality frontier has four points on it, and three of the four cost eleven cents or less per attempt.
-
Token price predicts almost nothing about task cost. Claude Sonnet 5 and Claude Sonnet 5.5 have the same list price. Sonnet 5 cost $1.19 per attempt; Sonnet 5.5 cost $0.07. The difference is entirely how long the agent works: 55 model calls per attempt against 8.
-
The harness changed the result for three models out of ten and made no measurable difference for the other seven. Where it mattered, it mattered a lot: Kimi K3 scored 10 points higher under Devin CLI than under OpenCode, and MiniMax M3 scored 12 points higher under mini-SWE-agent than under Claude Code.
-
Turning the reasoning-effort dial up did not pay. Across four models it raised cost per attempt by 1.1x to 2.6x and moved solve rates by between -2.5 and +3.5 points, none of it surviving correction for multiple comparisons.
-
Giving the agent the repository's own
CLAUDE.mdandAGENTS.mdmade it 58% more expensive on Opus 5.5 and no better at solving tasks. A minimal, hand-curated instruction pack was followed faithfully, cost 28% more, and also bought nothing. -
Two context-compression tools, Headroom and RTK, removed 7 to 9% of input tokens and did not change the bill by any measurable amount. The savings were consumed by extra model calls, and most of the cost they target is already discounted cache traffic.
Figure 1 shows every combination on one chart.
The top of the table is a tie
Figure 2 is the full ranking with 95% confidence intervals. The intervals are wide. Each cell is 600 attempts, but they are clustered in 200 tasks, and the task is the unit that varies. Two cells that differ by three points are not distinguishable here; a pair needs about 6.6 points of separation before the difference is unlikely to be chance, and the strictest multiple-comparison step asks for 11.5. Of the 666 pairs in the table, 153 are distinguishable after Holm correction.
SWE-2 finished first in this run; a rerun could easily reverse the order at the top. The more useful reading is in tiers. Seven cells sit between 69% and 72%: SWE-2 at both effort levels, Claude Opus 5 under two harnesses, GPT-6.1 Sol at both effort levels, and Gemini 3.7 Flash. Then a broad middle from 61% to 67% that holds most of the table, including every Claude Sonnet cell, Claude Opus 5.5, Kimi K3, GLM-5.2 and 5.3, Grok 4.6 and the GPT-5.6 models. Then a tail: Nemotron 3 Ultra, MiniMax M3 under Claude Code, and Claude Haiku 4.5.
Two other columns in Figure 2 matter as much as the solve rate. "Solved on all 3" is the share of tasks a combination got right every time it was asked, which is closer to what a developer sees when they run an agent once. SWE-2, GPT-6.1 Sol and Gemini 3.7 Flash all sit at 62 to 64% on that measure. Claude Opus 5 under OpenCode, which ties them on solve rate, drops to 59%. Kimi K3 under OpenCode solved 56% of tasks overall but only 38.5% every time, because 16% of its attempts ran out the clock.
The other column is median agent time. Claude Sonnet 5.5 finished a typical attempt in 26 seconds. Gemini 3.7 Flash under Devin CLI took 31 minutes for a solve rate six points higher. Both matter in daily use.
Four points on the frontier
The frontier in Figure 1 is short. GPT-5.6 Luna under OpenCode at $0.05 per attempt and 61.7%; GPT-5.6 Luna under Devin CLI at $0.07 and 64.5%; GPT-6.1 Sol under Claude Code at $0.11 and 69.7%; SWE-2 under Devin CLI at $1.48 and 71.7%. Everything else is beaten on both axes by something on that line.
The cost figures are mean cost per attempt at list rates, including attempts that failed. Cost per solve, which divides total spend by successes, stretches the gap: GPT-6.1 Sol delivered a passing change for $0.15; Claude Opus 5 under OpenCode for $6.08.
The frontier skips most of the cost axis. Between $0.11 and $1.48 there are fifteen cells, and none of them solves more than GPT-6.1 Sol does for $0.11. That is the case for routing: most of the work engineers hand to an agent does not need a $3 attempt, and on this task set a $0.11 attempt is not a compromise on quality.
Token price is a poor guide to task cost
Engineers tend to estimate a model's cost from its price per million tokens. On this benchmark that estimate can be off by an order of magnitude.
Claude Opus 5 lists at $5 in, $25 out. Claude Opus 5.5 lists at $4 in, $20 out, a 20% discount. Per attempt, Opus 5 cost $3.31 and Opus 5.5 cost $0.67, a 5x gap. Opus 5 made 63 model calls per attempt and read 3.6 million tokens of context doing it; Opus 5.5 made 22 calls and read 0.7 million. The two solve rates were 70.8% and 65.2%.
Claude Sonnet 5 and Sonnet 5.5 have identical list prices. Sonnet 5 cost $1.19 per attempt over 55 calls and a median of six and three-quarter minutes; Sonnet 5.5 cost $0.07 over 8 calls and 26 seconds. They solved 64.8% and 63.0% respectively.
Kimi K3 lists at $3.30 in and $16.50 out, 65% more per token than Sonnet 5. Its cost per attempt was $1.21, two cents more than Sonnet 5's.
The right-hand panel of Figure 3 is the relationship that actually holds: how many times the agent goes back to the model. That is a property of the model's working style and of the harness together, and it is why the same model under two harnesses can differ in cost by anywhere from 30% to more than double while solving the same share of tasks.
The harness: decisive for some models, invisible for others
We ran ten models under at least two harnesses each, on the same 200 tasks, and paired the results per task.
For seven of the ten, the harness made no measurable difference to the solve rate. For three it did. Kimi K3 scored 66.0% under Devin CLI, 61.7% under Claude Code and 56.0% under OpenCode; 97 of its 600 OpenCode attempts hit the time limit. MiniMax M3 scored 64.2% under mini-SWE-agent and 52.0% under Claude Code, where it gave up without changing anything in 96 attempts. Nemotron 3 Ultra scored 53.2% under Claude Code and 44.5% under OpenCode.
Cost was far more sensitive to the harness than quality was. Claude Opus 5 solved 70.8% under Claude Code for $3.31 and 69.2% under OpenCode for $4.21. Claude Sonnet 5 cost $1.19, $1.81 and $2.01 under Claude Code, OpenCode and Devin CLI for 64.8%, 62.2% and 61.7%. The direction is not consistent either: Nemotron was cheaper under OpenCode and worse; Kimi K3 was most expensive under the harness that scored it highest.
Two harness defaults cost us a restart during the OpenCode campaign and are worth knowing about if you run it headless. OpenCode keeps a workspace snapshot for undo, which on a 238,000-file repository ran its own git processes for long enough to time out 11 of the first 194 attempts; we turned it off. It also treats any path outside the workspace, including /tmp, as a permission prompt, and in headless mode nobody answers; agents that wrote test output to /tmp idled until the clock ran out. Both are configuration, not model behavior, and both would have been invisible in a leaderboard that reported only the final percentage.
Higher reasoning effort did not pay
Four models were rerun with reasoning effort set above the vendor default: Claude Opus 5.5, Claude Sonnet 5.5 and GPT-6.1 Sol at high instead of medium, and SWE-2 at max instead of high.
| Model | Default | Higher effort | Change (95% CI) | Cost per attempt |
|---|---|---|---|---|
| Claude Sonnet 5.5 | 63.0% | 66.5% | +3.5 (+0.2 to +7.0) | $0.07 to $0.18 |
| Claude Opus 5.5 | 65.2% | 65.5% | +0.3 (-2.3 to +2.7) | $0.67 to $0.99 |
| GPT-6.1 Sol | 69.7% | 69.0% | -0.7 (-3.8 to +2.3) | $0.11 to $0.27 |
| SWE-2 | 71.7% | 69.2% | -2.5 (-5.3 to -0.2) | $1.48 to $1.64 |
Sonnet 5.5 improved and SWE-2 got worse, each by a margin that is just outside zero before correcting for the four comparisons and inside it afterwards. Cost rose for all four, by between 10% and 160%. The vendor defaults are the better setting on this benchmark. Sonnet 5.5 at high effort does have one distinction: it was the most consistent cell in the entire run, giving the same answer on all three attempts for 91% of tasks.
Consistency, and what the tasks can tell apart
Across every cell, the three attempts agreed 79% of the time. The remaining 21% is why we run three and why a single-attempt leaderboard overstates its own precision. The spread between models is wide: 91% for Sonnet 5.5 at high effort, 68% for Nemotron 3 Ultra and Haiku 4.5.
Of the 200 tasks, 92 were solved by every model at least once, 15 by none, and 93 split the field. The 93 produce the ranking; the 92 show how high the floor is. Eleven tasks were solved by every model on every attempt.
The repository's own agent instructions: slower, not better
The second question was whether handing the agent the repository's own CLAUDE.md, AGENTS.md and .claude/rules files, the context a developer gets when they open Claude Code in the repository, improves results. Every run above was done without them. We ran two campaigns on Claude Opus 5.5 and Claude Sonnet 5.5, 200 tasks, three attempts, each condition paired against a bare arm on the same hosts.
The first campaign used each repository's instructions as committed, read from a revision before the task's fix landed so the files could not describe the answer. The second used a minimal pack we wrote ourselves: the build and test commands, nothing else, with a same-length placebo pack as a control.
Neither form of context moved the solve rate. The repository's own instructions moved Opus 5.5 by -1.0 points (interval -3.5 to +1.5) and Sonnet 5.5 by +1.8 (-0.8 to +4.7). The minimal pack moved them by 0.0 and +1.7. Cost went one way only: +58% for Opus 5.5 with the repository's own context, +28% with the minimal pack and +14% for Sonnet 5.5 with the minimal pack, all significant after correction. The placebo pack, which had the same length and no content, cost 4 to 6% more, so most of the increase came from what the instructions said rather than from their presence.
Figure 8 shows why. On the Java platform, whose AGENTS.md tells the reader to discover the relevant feature documentation and run the module's Gradle tests, the agent did exactly that. Tool calls that read AGENTS.md and the files it points to went from 3 across all 348 bare attempts to 1,511. Gradle invocations doubled. Time spent inside Gradle went from 6 to 18 minutes per attempt, and the hours agents spent polling a background build that had outlived its shell timeout went from 24 to 60. Gradle plus polling account for 97% of the extra wall-clock time. The two tasks that only the context arm solved were both fixed before the first build ran, so the builds changed no outcome.
A control arm of 31 tasks where the plumbing was in place but no context was delivered moved cost by 1.4%, so the mechanism is the content. In the second campaign we also confirmed that the pack was read: adherence to its instructions rose by 85 points for Opus 5.5 and 24 for Sonnet 5.5 relative to the bare arm. The agent followed the instructions; they told it to do expensive things that did not change the answer.
One caveat. The second campaign denied the agent network access, which the first did not, and offline Gradle builds that stalled on unreachable dependency hosts inflated part of the first campaign's +58%. Re-measured under the second campaign's conditions, the repository-context cost penalty on Opus 5.5 is +28%, still clearly positive. The recommendation is unchanged: for headless agents on these repositories, no context is the right setting today. The instructions in those files were written for people, and for an interactive session where a person is waiting, running the tests is the right call. For a headless agent graded on the result, it is cost without benefit.
Context compression: Headroom and RTK
The third question came from the token bill. Two open-source tools, Headroom and RTK, promise to cut it by compressing what the agent sends to the model: Headroom as a local proxy that rewrites each request, RTK as a Claude Code hook that condenses shell output before the model sees it. Both were tested the same way, as a paired arm beside plain Claude Code on the same tasks, hosts and time window, with Claude Opus 5.5 and Claude Sonnet 5.5, 1,200 attempts per tool per model. We set the adoption bar in advance: at least a 5% saving in cost per attempt, no solve-rate loss worse than 3 points at the lower bound of the interval, and no open reliability or security finding.
Neither cleared it. Headroom 0.39.1 took Opus 5.5 from $0.71 to $0.70 per attempt and Sonnet 5.5 from $0.07 to $0.07. RTK 1.0.1 took Opus 5.5 from $0.50 to $0.53 and Sonnet 5.5 from $0.123 to $0.124. Solve rates moved by -1.5 to +2.0 points, none significant, but the Opus intervals reached -3.3 and -4.0 at the lower end, which fails the solve condition on its own. Both tools ran cleanly: no proxy errors and no infrastructure failures across 4,800 attempts.
Figure 10 explains the arithmetic. Headroom did what it says: it removed 7.8% of input tokens per call on Opus and 8.5% on Sonnet, measured from its own proxy logs. But input is mostly cache reads, which cost a tenth of fresh input, so 8% fewer input tokens is worth 3 to 4% of the bill. And the agents took 5 to 9% more turns with it, each turn re-reading the whole cached context, which consumed the 3 to 4%. Reading the transcripts, we found the model noticing: in three of six attempts on two tasks, it described a code read as "elided" or "garbled" and read the file again. Headroom's own documentation says it pays off on long sessions with heavy tool output. Our sessions are short, a median of 18 turns on Opus, and in the shortest third of tasks Headroom made attempts measurably more expensive, by 14%. In the longest third it trended cheaper by 6%, not distinguishable from zero.
RTK's ceiling can be computed from its own counters. The hook fired in 99% of attempts and removed about 2,500 tokens of shell output per Opus attempt. Had every one of those tokens been written to the cache once and re-read on every later turn, the most RTK could have saved is 4.3% of the Opus bill and 3.5% of the Sonnet bill. That ceiling is below the bar before any behavioral effect is counted.
Both tools also change the agent's behavior in ways we did not set out to measure. RTK ships an instruction file that tells the model to treat condensed output as complete and to batch commands; Opus made 4% fewer calls with it. Headroom's default installation runs a telemetry beacon, checks PyPI for updates, downloads a model at runtime and reads the Claude Code credential store; we turned all of that off for the test, and would have to in production. Neither was adopted. Headroom's output shaper, which asks the model to be terse, is a different lever from input compression and remains untested.
Why we built our own
Public benchmarks are useful and we read them. They did not answer our questions for three reasons.
The tasks are public, so over time the solutions are in the training data, and a score stops meaning what it meant. Our tasks are mined from pull requests merged in the preceding weeks, in private repositories, and we retire a task and mint a replacement rather than edit it.
The code is not like ours. Most of the attempts above ran against a Java platform of 6.2 million lines of Java and 3.5 million of JavaScript across 238,000 files, with a Gradle build whose test cycle in some modules runs five to ten minutes. The other 84 tasks came from a TypeScript monorepo, two Python services and two Go services. Public suites are weighted toward mid-sized Python repositories. The agents that do well there do not all do well here, and the failure modes are different: a quarter of our failures on some cells were build breaks, not wrong answers.
And the thing we most wanted to measure was cost at the task level, under our harnesses, at our list rates, reconciled to our cloud meters. No public number does that.
How Touchstone works
Mining. Every merged pull request in a target repository is a candidate. The ticket it closed is the first source of the problem statement: the defect's steps to reproduce or the story's description, quoted verbatim, because text a human wrote before the fix existed cannot describe the fix. Ticket fields that could, such as engineering details and acceptance criteria, are never read.
Validation. A candidate becomes a task only if, at the parent commit, the pull request's test changes fail; with the full pull request applied, they pass; and they pass again on a repeat run. That proves the tests detect the change. It does not prove the change was a bug fix, and we deliberately admit enhancements too; a curator classifies each task by the kind of defect or change it represents.
Curation. A person reviews every task before it enters a release, and either accepts the quoted ticket text as the prompt or writes one. A curator who writes the prompt does so before seeing the pull request's own description, so the description cannot anchor it.
Two mechanical checks run on every save. A leak lint refuses a prompt that contains text from the reference fix. A second lint, which we added after the first campaign, refuses a prompt that omits a name the held-out test depends on. We found that when 57 tasks went unsolved by every model, 24 of them failed because the test asserted on an identifier, a class name, a message literal, a constructor signature, that the prompt never mentioned and that did not exist in the codebase the agent was given. Those were defects in the task, not failures of the model, and we re-minted 50 statements during that campaign before settling the release.
Sealing. Each attempt runs in a fresh snapshot of the repository at the task's base commit with no history, no remotes, no tags and a single synthetic commit. A 12-point audit checks every workspace before the agent starts. Databricks' post on their own benchmark describes finding agents that walked forward through git history to the merged fix; the seal exists so that walk has nowhere to go. We saw the same thing in a different form. Before we added process-level sandboxing in the first campaign, an audit of every tool call found nine attempts that reached outside their workspace, and one of them had read the reference fix from the host. All nine were discarded and rerun inside the sandbox, which now also hides the rest of the home directory. Since mid-October the agent has had no network access either.
Running. Every model-and-harness pair is run three times per task with a clean agent home, the harness's own steering files removed, and a per-task wall-clock limit that these campaigns ran at 2.5 times its released value. For models served through a shared gateway, each attempt gets its own virtual key so its spend is attributable. Rate-limit backoff is recorded per attempt; an unsolved attempt that lost two minutes or more to it is treated as tainted and rerun, and the report says how many were.
Grading. When the agent stops, any edit it made to a held-out test file is reverted, the real test patch is applied, and the tests run. The outcome is one of: solved, tests failed, build broken, gave up, timed out. Grader and infrastructure errors are a separate class that is rerun and never scored. Two rules came out of the first campaign. An agent that commits its own work moves HEAD, so the diff is taken against the sealed base rather than HEAD; three early attempts had been graded as "gave up" for committing a correct fix. And Claude Code, after a safety refusal from the provider on two tasks, silently finished the session on an older model; those attempts count as failures of the model that was asked for, and the report lists them.
Reporting. Every comparison is paired per task. Intervals are bootstrap intervals over tasks; significance uses a sign-flip test with Holm correction across every pair in the table; the report prints the minimum detectable difference so a reader knows what "no difference" can and cannot mean. Costs are list rates applied to the tokens each attempt recorded, and for the October campaigns the recorded tokens were reconciled against the cloud provider's per-deployment meters, which agreed to within 0.2 to 3%. Cells running through a bridge that did not report cache writes carry a stated ceiling of 10 to 15% on how far their true cost could exceed the figure shown.
Reading the numbers
A few things to hold in mind.
Everything is paired on the same 200 tasks, so cells are comparable to each other, but the absolute rates are a property of this release. A different mix of tasks would move every number together.
Costs are list rates. The Devin CLI cells are a bundled product with its own metering, converted at its list rate; enterprise discounts apply to some of the others. Timed-out attempts that recorded no usage are counted as attempts but not in cost, which makes a few cells' costs slight underestimates.
Wall-clock limits were 2.5 times each task's released limit. Timeouts are scored as failures, and the scoreboard says how many there were; the two cells with the most, Kimi K3 under OpenCode and Grok 4.6 under Claude Code, would look different with a longer clock, and we would rather report that than hide it behind a "pass among finished attempts" figure.
The main comparison pools campaigns run between early September and early October 2026. Harness versions moved during that window; each campaign's report records the one it ran under. Claude Code's --bare mode was used throughout the main comparison and the context and Headroom campaigns; RTK requires Claude Code's normal start-up, so its two arms were measured against each other and not against the bare cells.
What we take from this
Three conclusions follow directly. Routine agent work belongs on the frontier cells, GPT-6.1 Sol and the GPT-5.6 family, with SWE-2 and Claude Opus 5 reserved for the tasks that need them; the data does not support paying thirty times more for the typical task. Headless runs on these repositories should not load the repository's agent instructions. And neither compression tool is adopted, with a standing offer to re-test when a release changes what they do.
The benchmark itself keeps running. Every new model and harness goes through the same 200 tasks and the same report, and the task set is refreshed from newer pull requests over time so no model gets to train on it. Two things we want to measure next are plan-mode output, where the agent proposes rather than edits and a rubric rather than a test decides, and failure triage, where the agent is handed a failing build and asked what broke.
None of this needs anything a team with a test suite and a history of merged pull requests does not already have. The parts that took the longest were not the runner but the gates: proving a test detects the change, keeping the answer out of the prompt, keeping the answer out of the workspace, and refusing to report a difference the data cannot support.
