AI Agent Teams Burn Massive Compute for Minimal Quality Gains, Benchmark Reveals
New evaluations from Vals AI and Anthropic show that multi-agent architectures cost up to 5.1 times more than single agents while rarely improving output quality, mainly offering speed boosts offset by high coordination costs.
Deploying multi-agent AI systems has become a popular design pattern, but recent research shows that grouping models into teams often generates heavy compute costs with negligible performance improvements over a single agent.
According to evaluations conducted by Vals AI, agent teams consistently consume dramatically more tokens while delivering results that are barely superior to single-agent setups.
Little Advantage at Full Reasoning
Vals AI evaluated GPT-6 Sol and Claude Opus 5.5 across the "Vibe Code Bench," testing both models as solo agents and as collaborative teams under two operational settings: medium reasoning effort and maximum reasoning effort.
The multi-agent configurations drove costs between 1.8 and 5.1 times higher than their single-agent counterparts. However, across four direct comparisons between solo models and agent teams, only a single scenario demonstrated a statistically significant quality gain: GPT-6 Sol running at medium reasoning, where the team configuration outperformed the solo agent by 7.3 points.
When models operated at maximum reasoning effort, the collaborative setup gave neither GPT-6 Sol nor Claude Opus 5.5 any meaningful advantage. The findings indicate that when underlying frontier models already utilize full compute, assembling them into agent teams yields diminishing returns that fail to justify the steep token expenses.
Anthropic's Diminishing Returns
Anthropic's own internal testing with Claude Opus 5.5 showed a similar trend of diminishing quality returns as agent counts grew. While larger agent teams attained specific performance thresholds more quickly, scaling from 10 to 100 agents only marginally improved scores after 24 hours. In additional ProgramBench testing, speed gains consistently arrived alongside increased token consumption.
Benchmark scores across varying team sizes illustrate the plateau:
- Knowledge base task (Opus 5.5): 1 agent scored 0.53; 10 agents scored 0.70; 30 agents scored 0.71; 100 agents scored 0.74.
- Lean Theorem Proving task (Opus 5.5): 1 agent scored 0.39; 10 agents scored 0.66; 30 agents scored 0.66; 100 agents scored 0.68.
Anthropic also evaluated Fable 5.1, which displayed stronger relative quality gains on the Lean theorem proving task above 10 agents, though it still trailed Opus 5.5 across all tests. On the knowledge base evaluation, Fable 5.1's score dropped slightly when scaling the swarm from 30 to 100 agents.
Speed Gains and the "Coordination Tax"
Industry researchers point out that the primary practical benefit of multi-agent setups is task completion speed rather than enhanced intelligence. Speaking on the Dwarkesh Podcast, OpenAI researcher Noam Brown noted that multi-agent systems predominantly buy speed. In tests, four agents completed tasks twice as fast but incurred twice the compute cost; at 16 agents, this trend persisted but grew slightly less efficient.
Brown highlighted that parallelization depends heavily on the problem domain. While web research and mathematics can be divided among multiple models effectively, tasks like creative writing cannot. Deploying 10,000 agents to write a novel, Brown noted, is as ineffective as assigning 10,000 human writers to the same text. He added that running experiments with massive agent fleets remains largely uncharted due to prohibitive costs.
OpenAI developer Eric Provencher also cautioned against agent swarms, warning that inter-agent communication frequently falters as numbers grow. Provencher described this inefficiency as a "coordination tax," arguing that without strict architectural necessity, complex agent swarms often represent wasted money.
