NVIDIA reports that its experimental Noah framework reached 82.2% on SWE-bench Verified using about 28 model calls and 1.1 million tokens per task. A rival harness scored four points lower with 66 calls and 2.2 million tokens.
The efficiency result may matter more than the score. It indicates that agent performance depends partly on how the surrounding system prompts the model, manages state, and permits actions. But NVIDIA’s benchmark claims have not been independently reproduced. This remains a reported result, not a settled one.
The harness changes the performance equation
Noah turns typed Python classes into agents. Docstrings become prompts, object fields hold state, and generated Python controls the workflow.
This construction may explain how the framework completes the same class of work with fewer calls and tokens. The model is only one component. Prompts, state management, available tools, and execution logic all affect the cost and quality of the result.
Better orchestration also creates new control obligations.
Valid output is not authorized action
Noah can use type checking to confirm that an output has the expected shape. That does not establish whether the requested action was authorized. A structurally valid instruction to issue a refund is still unsafe if the agent lacks permission to approve it.
NVIDIA calls Noah an “open experimental surface.” Its package was at version 0.0.8, and NVIDIA explicitly says its validators are not a containment boundary. Because the framework executes generated Python, it requires isolation through a container or virtual machine, careful permission design, authorization checks, and auditing.
The same harness that improves efficiency becomes another source of failure and control. A clean abstraction does not remove operational risk.
Allocated context is not useful context
Ornith 1.5 35B shows a similar gap between headline capability and usable performance.
In a documented RTX 4070 Ti test, the model generated 50 to 56 tokens per second and processed 650 to 700 prompt tokens per second. That supports the claimed generation rate of 53 tokens per second on similar hardware. The headline figure of 155 tokens per second on an RTX 4090 is less secure because no source was found for it.
Its 21.71 GB Q4KM build can barely fit on an RTX 4090 alongside a quantized context cache. Fitting 170,000 tokens does not show that the model remains useful across that context.
Ornith 1.0 reportedly degraded at around 70,000 to 80,000 filled context tokens. No equivalent Ornith 1.5 test had been published, and its SWE-bench Verified score of 79 remains a vendor measurement.
Capacity is not the same as demonstrated usefulness.
Convenience layers become control layers
OpenRouter offers unified billing, routing, and failover across a large collection of models and providers. Stripe’s purchase of the company places payment infrastructure, metering, routing, and commercial relationships inside one company.
The layer that simplifies access also becomes a concentrated point of control.
The Grockbot account offers a practical response through workflow sketches involving email triage, research, newsletter drafts, website deployment, and delegation to a local model. These examples are not independent evidence of productivity. Their strongest pattern is constrained access: email responses remain drafts, credentials are entered privately, permissions are limited, and humans review the work.
Evaluate the system delivering the work, not the headline capability. Measure resource use, useful context, action boundaries, and auditability together.
Controls do not make an agent reliable. They limit the damage when it is not. The relevant unit of evaluation is the complete system under independent testing.