Elaborate AI Agent Harnesses Triple Token Costs Without Improving Coding Scores, Study Finds

Makers of AI developer tools frequently claim that sophisticated agent harnesses significantly improve coding benchmark scores. However, a recent empirical study titled "What does a harness buy? Mostly tokens." reveals that complex harnesses often do little more than inflate the token bill.

Conducted by researchers Liu and Han at Shandong University, the study measured the real-world impact of agent harnesses—the system prompts, specialized tools, and context-management layers that enable large language models (LLMs) to navigate and edit code repositories. Their findings indicate that swapping between frameworks alters benchmark success rates no more than running the exact same configuration twice, even while driving inference costs up by as much as 300 percent.

Measuring Noise Instead of Progress

Prior benchmarks and vendor claims frequently report margins of over 20 points between different agent frameworks. According to the researchers, these claims almost always rely on single benchmark runs.

On SWE-bench Verified, running identical setups multiple times can naturally produce score swings of up to six points purely due to measurement noise. For instance, Zhipu AI previously claimed a 2.4-point advantage for its proprietary harness over Claude Code—a difference that falls well within the expected variance of a rerun. Furthermore, varying benchmark integration methods across vendors make direct leaderboard comparisons unreliable.

To isolate the specific effect of the harness, the researchers evaluated five models across shipped versions of three frameworks:

• Claude Code: Features specialized file and search tools, automatic history compression, and a system prompt with tool definitions totaling roughly 16,600 tokens.
• OpenCode: Features specialized tools and an initial prompt overhead of around 7,000 tokens.
• mini-SWE-agent: A bare-bones setup providing only a standard Bash tool, no context management, and a system prompt under 1,000 tokens.

The evaluated models included Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash, and HY4-Preview, with Claude Opus 5 tested as a reference within Claude Code. Evaluations covered the 45 hardest tasks on SWE-bench Verified alongside an expanded pool of 447 tasks, with a cross-check on Terminal-Bench 2.1.

All evaluations were conducted entirely offline. During initial tests with internet access enabled, 71 percent of attempts by Deepseek and Qwen models retrieved fixes directly from GitHub rather than solving the problems autonomously. While the authors did not allege training on test data, web access was permanently disabled to maintain benchmark integrity.

Swapping Harnesses Behaves Like a Coin Toss

On the broad sample of 447 tasks, Claude Code (the heaviest harness) and mini-SWE-agent (the leanest) finished within two points of each other across the tested models—statistically indistinguishable within a five-point confidence band.

When analyzing individual problem outcomes:

• Rerunning the same setup flipped the median outcome on 13 percent of tasks.
• Swapping harnesses flipped the outcome on the exact same 13 percent of tasks.
• Swapping underlying models shifted outcomes on 22 percent of tasks.

On the 45 hardest tasks, frameworks diverged by just two to five solved tasks per model, while an identical rerun shifted the count by up to three. The authors describe the performance variance between harnesses as behaving like a "fresh draw of the same noise."

The only framework that showed a statistically significant deficit on the 447-task pool was OpenCode, which trailed the others by up to nine points. The researchers traced roughly half of this gap to output limit handling: when a model hits an output ceiling mid-step during reasoning, Claude Code prompts it to continue, whereas OpenCode terminates the step, resulting in task failure. OpenCode also suffered from tighter time limits on local models and search tools that broke offline due to initial binary download attempts.

The Cost of System Prompts

While pass rates remained essentially flat, token consumption diverged sharply. Because harnesses resend system prompts and tool descriptions with each conversational step, initial prompt length dictates baseline cost:

• Claude Code: Resends roughly 16,600 tokens of system instructions and tool schemas per step, while also preserving reasoning traces in conversation history.
• OpenCode: Resends approximately 7,000 tokens per step.
• mini-SWE-agent: Resends under 1,000 tokens per step.

For GLM-5.3-Flash, a single task attempt cost 1.81 yuan in Claude Code compared to 0.56 yuan in OpenCode. With Deepseek-V4-Flash, mini-SWE-agent required additional interaction steps, bringing its total token cost roughly even with Claude Code despite its minimal prompt overhead.

Additionally, Deepseek's proprietary harness scored no higher with its own model than competing frameworks did, while recording the highest cost. Claude Opus 5 also delivered identical benchmark results even when researchers stripped away its specialized tools in Claude Code.

The Role of Reasoning

The only architectural factor that consistently moved benchmark scores beyond noise was internal reasoning ("thinking"). When reasoning was disabled, mini-SWE-agent paired with Qwen3.6 lost 11 points, while HY4-Preview in Claude Code also dropped by 11 points. However, the degree to which an agent depends on reasoning appears tightly coupled to specific model-framework combinations.

The researchers conclude that a harness primarily buys "fewer ways to lose a task, and a bill, not pass rate." To prevent misleading marketing claims on leaderboards, the authors urge developers and benchmark organizers to publish repeated runs of identical configurations alongside any comparative harness evaluations.