KuaiPilot found that sandbox failures caused around 16% of incorrect punishments while evaluating Kuaishou’s 35-billion-parameter coding model. After changes to the evaluation system, that rate fell below 2%.
The model reportedly scored 69.4 on SWE-bench Verified, but the score alone hides a critical part of the system. A faulty grader can punish a model for errors it did not make. It can also teach the model from false failures.
The evaluation environment is part of the result.
The evaluator can change the result
The same Kuaishou model scored 32.6 on Terminal Bench under one agent and 49.4 under another. Separate scaffold comparisons moved model results by as much as 48 points.
Differences that large make it difficult to treat the harness as an implementation detail. The score reflects a combination of model weights, agent behavior, tools, prompts, execution structure, and grader quality.
The model is one component of the evaluated system, not the sole cause of the reported result.
Use only as much structure as the task needs
This does not mean every model needs the most elaborate agent system available.
Simple loops worked well for bounded tasks whose results could be verified cheaply. Explicit graphs helped with long-running or high-stakes workflows by providing checkpoints and constraining error propagation.
The right question is how much execution structure the task needs.
RAmjad’s browser verification setup used isolated E2B sandboxes, Playwright recordings, database seeding, and a judge agent. The important output was evidence. More autonomy did not establish that the agent completed its task correctly. Verification did.
Speedups belong to configurations
Multi-token prediction delivered reported gains as high as 3.11 times on Gemma 4 31B. That result depended on acceptance rate, draft depth, context length, and concurrency.
In one test at 120,000 tokens of context, speculation was slower than plain decoding. The extra work helped under suitable conditions and hurt under others.
A speedup is not a permanent property of a technique. It belongs to a workload and configuration.
Hardware rankings depend on deployment fit
One reported comparison showed a 3.8 times Kimi K3 advantage on MI355X over B200, with memory capacity and node topology playing major roles. When the comparison shifted to the equally capacious B300, the B300 node reportedly delivered about 1.65 times the throughput.
Those results do not establish a universal silicon ranking. They suggest that deployment fit, including memory capacity and topology, can dominate the headline comparison.
Moonshot’s six-times decoding claim also came from a 48-billion-parameter development model, not a full-scale K3 comparison. It may describe that development setup accurately, but it does not establish the same advantage for the full model.
Before trusting a benchmark or speedup, inspect the complete system that produced it.
Simple rankings become less comfortable once the full system is visible, but the evidence becomes more credible. Treat every score and speedup as a system result.