Real-World Benchmark Shows AI Coding Agents Fail to Finish Nearly 75% of Tasks

While conventional evaluation suites suggest that AI coding agents have essentially mastered programming, a new test based on messy, real-world user requests reveals a starkly different reality. Even the highest-performing frontier models leave tasks incomplete roughly three out of four times, requiring human software engineers to step in and finish the job.

Researchers from Alibaba and several Asian universities introduced Co-Design Bench, a benchmark designed to evaluate how coding agents handle realistic software modifications. Unlike older benchmarks where agentic models routinely score above 99% by running, testing, and fixing their own code, Co-Design Bench exposes a gap of more than 35 percentage points between current agent performance and true task completion.

Real-World Messiness Exposes Benchmark Saturation

The benchmark targets web application revisions based on user instructions, which often incorporate multimodal inputs such as images, audio recordings, documents, or videos. Rather than evaluating isolated programming puzzles, the team pulled tasks from actual user coding sessions, deliberately prioritizing interactions where human developers previously had to rephrase their prompts multiple times or roll back broken changes.

Tasks in the benchmark span from minor adjustments—such as extracting a numerical amount from a receipt and updating all dependent totals—to full-scale project refactoring. Evaluation is rigorous: a primary AI judge tests the running application inside a live browser, after which a secondary judge checks the result to rule out mock-ups or hardcoded cheat values. Credit is granted only if the requested feature genuinely works in the live interface.

Under these conditions, top-tier models struggled to deliver end-to-end solutions:

• GPT-6 Astra took first place among the ten models tested, satisfying roughly 70% of individual requirements on average. However, it achieved fully correct execution on only 26.4% of tasks.
• Claude Opus 5 and Qwen3.8-Max claimed the next spots on the leaderboard.
• Grok 4.6 ranked last among the evaluated frontier systems.
• Qwen3.8-27B, a substantially smaller model, climbed to second place specifically on larger repository overhauls, though researchers noted this did not indicate a general advantage across all coding domains.

Multi-Faceted Demands Overwhelm Agents

The benchmark highlighted a clear bottleneck: performance deteriorates rapidly as tasks combine multiple types of revisions, such as simultaneous changes to layout, user interaction, visual design, and underlying content. While this multi-layered complexity is standard practice in production software development, every tested model showed steep performance drops when three or four change categories were present at once.

Throwing more computational resources at the problem did not resolve this shortfall. For example, GLM-5.2 consumed nearly six times as many tokens as GPT-6 Astra on smaller change requests while still achieving significantly lower scores. Even GPT-6 Astra fully solved only 41% of multimodal track tasks and dropped below 18% on broader repository-level tasks.

The Software Harness Matters as Much as the Model

The study also demonstrated that model benchmarks tell only part of the story. When researchers tested the same AI models using different harnesses—the underlying software scaffolding that governs how an agent inspects files, edits code, and executes terminal commands—outcomes shifted dramatically.

In one instance, Qwen3.8-27B scored 75% in one harness environment but plummeted to 56% in another on large overhauls. In several cases, changing the execution harness altered which model outperformed the other, prompting the authors to recommend that engineering teams evaluate models and their operational harnesses as an inseparable pair.

The findings align with broader industry observations regarding AI evaluation gaps. Prior research by the evaluation group METR found that real-world pull request acceptance rates by human project leads trailed scores on SWE-bench Verified by approximately 24 percentage points. While Co-Design Bench still has constraints—such as testing static, single-turn prompts without mid-task dialogue—it demonstrates that coding agents remain powerful assistants rather than autonomous developers capable of delivering production-ready features unattended.