How can the exact same artificial intelligence model, evaluated across the exact same task suite, score 73.9 under one execution scaffold and drop to 65.8 under another?
That 8.1-point gap was recorded when Google’s Gemini 3 Pro was evaluated on Terminal Bench 2.1. Wrapped in the specialized Terminus 2 agent harness, the model achieved 73.9. When executed through Google’s default Gemini CLI, the score plummeted to 65.8. The model weights, task definitions, and grading rubrics remained completely identical.
The divergence demonstrates that prompts, tool schemas, retry heuristics, and context compression rules cannot be dismissed as secondary implementation details. In production agent architectures, engineering rarely eliminates an operational constraint entirely. More often, it shifts the constraint into another layer of the stack: memory pressure moves to disk latency, expanding tool catalogs degrade routing precision, and fast unverified generation creates downstream verification debt.
Harness Engineering Moves Leaderboards by Eight Points
The Terminal Bench evaluation proves that foundation models are only raw reasoning engines. How that reasoning translates into terminal execution depends on the software scaffold enclosing the model.
Terminus 2 outperformed the default CLI by structuring how terminal errors, directory listings, and tool outputs return to the context window. When a command fails, a naive harness dumps raw stderr into the prompt, often confusing the model with verbose tracebacks. An optimized harness intercepts known failure signatures, normalizes path discrepancies, and presents structured correction cues.
This distinction explains why public benchmark leaderboards can mislead enterprise engineering teams. A model that appears mediocre in a basic conversational wrapper may perform exceptionally well when paired with disciplined middleware. Conversely, a high-scoring model will stumble if deployed inside an unoptimized shell wrapper.
Pruning Context Triples Completion Rates While Halving Runtime
While unmanaged context causes agent failure, disciplined context management can improve performance, execution speed, and token cost simultaneously.
Microsoft published findings from an evaluation of a 50-task autonomous agent benchmark. By applying structured conversation pruning and intermediate summarization across multi-step tasks, the engineering team recorded remarkable compounding efficiencies:
- Total token consumption dropped from 1.48 million tokens down to 553,000 tokens, a 63% reduction.
- End-to-end workload runtime fell from 14.6 hours down to 5.8 hours.
- Successful task completion increased from 71.0% up to 91.6%.
This evaluation dismantles the assumption that context management is merely a cost-reduction tactic. Truncating stale conversation turns, stripping resolved debugging attempts, and summarizing completed milestones directly improved agent reasoning. Long context windows suffer from attention dispersion, where models overlook critical instructions buried beneath thousands of tokens of historical tool logs. Keeping context lean allowed the agent to stay focused on immediate operational goals.
However, pruning policies must match task semantics. Stripping context too aggressively risks deleting environmental variables, authentication states, or error codes that the model needs three turns later. Summarization policies must be designed around workflow requirements rather than applied as blunt truncation rules.
Tool Bloat Degrades Agent Routing by 23 Points
While developers often assume that providing agents with more capabilities improves utility, expanding tool catalogs introduces severe routing friction. A study analyzing an enterprise assistant managing 110 specialized agents and 584 registered tools revealed that routing accuracy on ambiguous user requests dropped by 16 to 23 percentage points as the tool registry expanded.
When an agent is presented with hundreds of overlapping tool schemas, parameter descriptions begin to blur. Given an imprecise query, the model frequently misroutes requests to inappropriate APIs, triggering cascading failures.
These results strongly favor small, task-specific tool surfaces over monolithic registries. A registered tool that is routed incorrectly represents negative capability. Production architectures should dynamically filter tool definitions based on active workflow state, exposing only the three to five operations relevant to the immediate subtask.
Moving Hardware Bottlenecks: Prefill Speeds and 32-Second Tokens
The rule that optimization moves constraints rather than erasing them is most visible in hardware deployments. An open-source implementation of Moonshot AI’s Kimi K3 achieved byte-identical generation across memory budgets of 8, 32, 95, and 224 GB.
Superficially, running a multi-trillion parameter model in 8 GB of memory sounds revolutionary. In reality, the 8.24 GB configuration required 1.7 TB of free high-speed NVMe storage and generated exactly one token every 32.69 seconds. The memory constraint was not eliminated; it was simply transferred into massive storage allocation and unusable latency.
Workload topology also dictates which hardware metrics matter. On an RTX 3060 coding workstation running llama.cpp, prefill throughput reached 2,137 tokens per second, while generation hovered at 75 tokens per second. Because software engineering agents spend the vast majority of their compute cycles re-reading codebases, project diffs, and test suites, prefill bandwidth dominates the developer experience far more than raw generation speed.
Software optimization can also completely rewrite silicon capabilities. SemiAnalysis demonstrated that an advanced kernel dispatch overhaul increased AMD MI355X peak throughput on Kimi K2.5 from 348 to 2,687 tokens per second per GPU. The hardware was unchanged; the execution stack unlocked an eight-fold increase.
Assessing AI systems requires evaluating the entire operational pipeline. Engineers must measure where constraints shift, whether into token volume, storage bandwidth, inference latency, routing accuracy, or human review.