Open AI Models Close Cyber Capability Gap with Closed Frontier Systems at Fraction of Cost
New benchmark results from research firm Vals AI show open-weight language models rapidly narrowing the cybersecurity capability gap against top closed models, reaching complex attack objectives at significantly lower token costs.
The capability gap between open-weight and proprietary closed AI models in autonomous cybersecurity tasks is shrinking rapidly, according to new benchmark data from research firm Vals AI.
In tests conducted on the Vals CyberOffense benchmark, freely available open-weight models approached the performance of top-tier closed systems while executing multi-stage network attacks at a tiny fraction of the operating cost.
Autonomous Corporate Network Exploitation
The Vals CyberOffense evaluation tests AI agents on isolated attack environments simulating realistic corporate networks. Each model operates solo on an isolated Kali Linux machine with zero human aid, starting with only a public web address. To succeed, the AI agent must issue command-line instructions, breach web applications, navigate Windows servers, and escalate privileges to gain full administrative control. All commands and actions are externally logged and automatically scored across two main scenario suites: Nova (21 objectives) and Lighthouse (31 objectives).
In the benchmark tests, GPT-6 Sol Daybreak—a model developed under OpenAI's restricted Daybreak cyber program—achieved the top score overall, reaching 80.8 percent of attack objectives. In the multi-company Lighthouse test, Daybreak successfully took over administrator control of an initial company, harvested passwords, and pivoted into a second company's network, completing 27 out of 31 objectives in under three hours.
However, the open-weight model Deepseek V4.1 Flash 1017 followed closely behind, achieving an overall objective completion rate of 72.8 percent. On the Nova test, Deepseek actually outperformed Daybreak, averaging 16.3 completed objectives (range 15–18) compared to Daybreak's 15.7 (range 12–19).
Crucially, Deepseek executed its runs at roughly $3 each, compared to approximately $22 per run for GPT-6 Sol Daybreak.
| Model | Nova Score (out of 21) | Lighthouse Score (out of 31) |
|---|---|---|
| GPT-6 Sol Daybreak | 15.7 (avg) | 27 |
| Deepseek V4.1 Flash 1017 | 16.3 (avg) | 21 |
| Kimi K3.1 Pro | 13 | 20 |
| Meta Wonder | 7 | 28 |
| Grok 4.7 | 12 | 16 |
| Kimi K3 | 9 | 10 |
| GLM 5.3 Flash | 5 | 5 |
Meta's open model, Meta Wonder, showed particularly strong performance on the more complex scenario, completing 28 out of 31 objectives on the Lighthouse test despite a lower Nova score of 7.
Extended Token Runtimes Reveal True Capabilities
A key finding from the research is the impact of compute duration on open model performance. Short benchmark evaluations frequently underestimate open models because their attack progress requires longer context processing.
During testing, Deepseek V4.1 Flash scored zero objectives within its first 25 million tokens of processing. The model only began achieving significant breakthroughs after crossing the 100 million token mark. Researchers noted that most open Chinese models tested did not make meaningful progress until passing this 100 million token threshold, highlighting that short-duration safety benchmarks miss the operational potential of these systems.
Shrinking Timeline and Cost Asymmetries
The Vals AI findings reflect a broader trend confirmed by public safety organizations. An analysis published in July by the UK AI Security Institute (AISI) noted that the capability gap between open-weight and closed frontier models in cyber domains had narrowed to between four and seven months, down from six to ten months across most of 2025.
Cost disparities between open and closed offerings remain stark. According to AISI evaluation data, running a cyber-range test with Deepseek V4-Pro costs $1.19, compared to $85 for Anthropic's Opus 4.5. Furthermore, safety guardrails embedded in open-weight models proved largely ineffective once weights were accessible, as users face no central provider restrictions on command execution.
Complementary findings from Anthropic revealed that the freely accessible GLM-5-3 model can construct security exploits at capability levels nearly matching Anthropic's internal frontier model, Mythos.
Defensive Implications
Vals AI researchers stressed that once a model's weights are published, providers lose the technical capability to control or revoke access to dangerous capabilities. Consequently, real-world network attacks using automated AI agents are anticipated to rise in the coming months.
However, the researchers pointed out that the trend offers an essential opportunity for cybersecurity teams: the identical open-weight models capable of probing corporate networks can be deployed locally by security defenders to audit systems, discover zero-day vulnerabilities, and patch security flaws before malicious actors can exploit them.
