How can a tool advertise up to 90% less Bash output while a controlled test finds 7.6% higher median task cost and 13.8% more turns?
Context management, reusable workflows, and compressed outputs can make an agent look more efficient. But fewer tokens or smaller outputs do not automatically mean lower cost, preserved quality, or reliable results.
Smaller Outputs Do Not Guarantee Lower Task Cost
RTK advertises up to 90% less Bash output. In JetBrains’ controlled Skills Bench test, however, median task cost increased 7.6%, the number of turns increased 13.8%, and quality was largely unchanged.
These measurements are not contradictory. Output compression can help with noisy commands, but it is not evidence of equivalent invoice savings. If the agent needs more turns to complete a task, the surrounding loop can consume the benefit.
Efficiency Can Cost Answer Quality
A preprint comparing codebase-memory-mcp with file reading reported that tokens per question fell from about 10,000 to 1,000, while tool calls fell from 4.8 to 2.3. Answer quality also fell from 0.92 to 0.83.
The system used roughly one-tenth the tokens and fewer than half as many tool calls, but produced lower-quality answers. Whether that trade-off is acceptable depends on the workload. These results do not support claiming that the optimization improved everything at once.
Benchmarks Measure the Model and Its Harness
The ten highlighted AI repositories focus on context, tools, permissions, execution, and review. None ships model weights or a training run. Their work controls what a model can read, change, execute, and expose to reviewers.
Separate Terminal Bench results changed substantially when the harness changed. That suggests a benchmark score can reflect both the underlying model and the agent loop around it.
Model capability still matters. Qwen 3.6’s model card reports 77.2 on SWE-bench Verified, compared with 80.8 for Claude Opus 4.6. The reasonable conclusion is that model capability and harness design both matter, while a benchmark headline may not reveal how much each contributed.
Reusable Workflows Need Safeguards
Claude’s Record a Skill can turn a demonstrated procedure into a reusable workflow, but no reliability evaluation was supplied.
Open Worker takes a more explicit approach. It classifies actions as reading, local writing, execution, or external activity. When unattended mode needs clarification, it waits in an inbox. This prevents the agent from silently guessing when the next action requires information it does not have.
Popularity Is Not Reproducibility
The ten repositories have 170,871 stars in total, but popularity does not establish that their claims are reproducible.
T3 MP3ST is the only repository in the group that provides a command for recomputing its benchmark claims from committed data. That gives users a way to inspect whether the reported result follows from the available evidence.
Treat the harness as part of the system being evaluated. Require separate evidence for cost, quality, and reliability, because improving one does not prove improvement in the others.