JFrog reviewed 55 submitted vulnerability advisories and found that 54 were fabricated. Cheap discovery has little value when verification cannot keep up.
That problem sits at the center of open-model engineering. Models are becoming credible tools for coding, software control, and vulnerability discovery, but useful capability belongs to the complete system: model weights, harness, memory economics, evaluation method, hardware, and verification process.
Discovery is only useful when verification can keep up
GLM 5.3 retains GLM 5.2’s 753 billion parameter base, with additional post-training focused on engineering and vulnerability work.
Z.AI reports 2,436 reviewed and deduplicated findings across 269 open-source projects. The reported severity totals do not align cleanly. One account says 197 findings were critical or high. Another says 1,097 were medium to high. These categories are different and should not be treated as interchangeable.
Still, the reviewed findings suggest that useful security work is not merely hypothetical. They also show why the verification process matters. Generating a possible vulnerability is only the beginning. The practical advantage goes to organizations that can triage and verify findings without being overwhelmed, not necessarily those generating the most.
GLM 5.3 shows progress without settling the rankings
GLM 5.3 scored 84.5 on Cyber Gym. On Exploit Bench, it scored 54.4 against 78 for the comparison model. On Exploit Gym, the result was 105 tasks against 181.
These results suggest that vulnerability detection may be approaching the comparison models faster than full exploitation. More importantly, GLM 5.3 increased its Exploit Bench score from 24.4 to 54.4 without changing the base weights. Better post-training appears to have produced a substantial improvement in that reported evaluation.
The relative rankings remain difficult to interpret:
- Z.AI ran some competing models inside its own harness.
- Cyber Gym warns that modest score differences may be stochastic.
- One account disputes a comparison labeled “Fable 5 with fallback,” arguing that it actually uses Mythos 5.
- Public leaderboard results prevent a simple declaration about which model is best.
The results may represent genuine progress, but the model cannot be separated from the conditions under which it was evaluated.
The harness changes performance and cost
DeepSeek Harness uses an append-only typed event log. Previously sent content never changes, and each request is reconstructed from recorded events. This preserves prefix caching even after conversation compaction.
That architecture has a direct cost consequence. The cited DeepSeek pricing puts cached input at about one third of a cent per million tokens, compared with 43.5 cents for uncached input. A harness that mutates earlier history can destroy that caching advantage.
Memory architecture is part of operating cost, not a minor implementation detail.
The harness can also affect measured capability. In one cited benchmark, identical Fable 5 weights scored 83.8% under Claude Code and 80.4% under Terminus 2. The comparison does not isolate the cause, but it supports treating scaffolding as part of the performance claim.
When identical weights produce different results under different harnesses, the score describes the complete evaluation system, not just the model.
Local capability ends where the memory budget does
QN reports that QN 3.8 27B beats Opus 4.6 Max on several coding-agent and software-control benchmarks, while losing on Terminal-Bench 2.1, GPQA Diamond, and HLE. QN ran both sides of the comparison, and the exact scores had not been independently reproduced on a named third-party harness.
Hardware further complicates the headline results. The model’s BF16 weights occupy 54.66 GB. A 24 GB GPU is described as the recommended practical tier. At 16 GB, tighter quantization and cache management are required. At 12 GB, the quality compromises become larger. The 8.5 GB 1-bit build leaves too little working memory for practical use.
A model can report strong benchmark results and still be a poor local tool if quantization, cache pressure, or limited working memory removes too much of its practical value.
Evaluate the system you can actually run and verify, not the model score in isolation.
The useful unit of evaluation is the model plus its harness, memory budget, evaluation method, hardware, and the people responsible for checking its work.