Gemini 3 Pro scored 73.9 on Terminal Bench 2.1 with Terminus 2 and 65.8 with Google’s Gemini CLI. The model and tasks remained fixed, but changing the harness changed the result by 8.1 points.
Prompts, tools, retries, and context handling are not incidental implementation details. They are part of the evaluated system.
The Harness Is Part of the Result
AI engineering rarely removes a constraint completely. More often, the constraint moves.
Memory pressure can become storage and latency. A larger tool catalog can become a routing problem. Faster generation can create more verification work for the recipient.
The Gemini 3 Pro comparison makes this visible. A model score does not describe the model alone when the harness can move that score by 8.1 points. The measured result belongs to the complete configuration.
Context Management Can Improve Reliability
Microsoft reported that pruning and summarization reduced a 50 task agent workload from 1.48 million tokens to 553,000. Runtime fell from 14.6 hours to 5.8 hours, while task completion increased from 71% to 91.6%.
This was not merely a cost reduction. The workload used fewer tokens, completed faster, and achieved a higher completion rate.
The result does not establish that the same policy will produce equivalent gains elsewhere. Remove the wrong information, and an agent may lose context it needs. The engineering question is whether a specific summarization policy preserves the information required by a specific workflow.
Added Capability Creates New Constraints
Research on an enterprise assistant with 110 agents and 584 tools found that routing accuracy on vague requests fell by 16 to 23 points as the tool catalog grew. More tools expanded what the system could potentially do, but made selecting the correct route harder when requests were unclear.
This favors smaller, task specific tool surfaces over treating integration count as a capability score. A tool that exists but is routed incorrectly is not useful capability.
The same trade-off appears in infrastructure. A Kimi K3 implementation produced byte identical output with memory budgets of 8, 32, 95, and 224 GB. But the 8.24 GB configuration required 1.7 TB of free NVMe and generated one token every 32.69 seconds.
The output was preserved, but the constraint moved from RAM to storage capacity and latency. The configuration demonstrated feasibility under tight memory limits. It did not make the setup practical on ordinary hardware.
Measure the Workload and Serving Stack
Throughput numbers depend on which part of the workload matters. An RTX 3060 coding agent build emphasized prefill rather than generation because agents repeatedly process conversation history, files, and tool output. In the cited llama.cpp benchmark, the same run reached 2,137 prefill tokens per second and 75 generation tokens per second.
For an agent that spends substantial time rereading context, generation speed alone cannot describe the experience.
Software can also change apparent hardware performance. SemiAnalysis reported that a dispatch change raised MI355X peak throughput on Kimi K2.5 from 348 to 2,687 tokens per second per GPU. An accelerator comparison may therefore measure the interaction between hardware, dispatch, model, and workload, rather than an isolated property of the accelerator.
Measure the complete workflow, then find where the cost moved.
That workflow includes the people receiving the output. Nate B. Jones argues that fast generation can transfer work rather than eliminate it. If generated material contains unclear prose or unverified claims, the recipient may need to check those claims, request revisions, and determine what the sender meant.
Drafting became faster. Verification and clarification did not disappear, and responsibility remained with the sender. Polished output is not evidence of accuracy. Faster generation does not remove the need to verify the work.