In an 18-game StarCraft tournament, not a single reasoning model played beyond a beginner level. Grok burned over 11,000 reasoning tokens across 43 minutes, issued six batches of commands, and never built a fighting unit. They got destroyed while they were thinking.
StarCraft Exposed the Real Cost of Generating Text
Ben Swerdlow, one of the founders of Freestyle, built an agent-only version of Brood War as an experiment. Every model configuration played every other configuration in a full round robin, with 18 games apiece. Swerdlow reported that “none of the models played beyond a beginner level. Not the winner, not the expensive ones, none of them.” He pointed out that “a beginner doing a photon rush, the cheap cannon cheese that anyone learns in their first week, would beat every single one of these agents.”
Tournament records showed a stark divide. Codex Astra, on its highest effort setting, went 18 and 0. Claude’s fable model went 15 and 3. At the bottom, Claude’s smallest model and Grok on low effort went 0 and 16.
The failure came down to latency against an active clock. StarCraft has a running clock where agents had one tool to read the game state and another to issue orders. Between those steps, they generated reasoning text. In Swerdlow’s words, “they got destroyed while they were thinking.” In that 43-minute match, Grok burned all those tokens on its highest effort setting just to issue six batches of commands without building a unit.
Actions per minute expose the mechanical problem. DeepMind capped its own StarCraft AI at roughly 260 actions a minute to keep it fair against human professionals. DeepMind’s AlphaStar reached Grandmaster in StarCraft II in 2019, the top league on the ranked ladder against human players. AlphaStar mapped visual state directly to actions without reasoning through sentences first.
By contrast, language models generating reasoning text fell far behind. Codex on high effort averaged around 13 actions per minute. Grok on high effort averaged under three.
Game costs were just as lopsided. The 18 and 0 winner cost about 10 and 1/2 dollars per game. The cheapest agent cost around 30 cents per game and went 0 and 16.
Probability Distributions Instead of Sentences
If generating text under an active clock breaks down, what happens when an agent stops generating sentences altogether? That question sits at the center of Jev. Type Safe came out of 2 years of stealth on September 15th with 40 million in funding. Diogo Almeida, who was on the original team that built ChatGPT, argues that superhuman chat did not deliver what it promised, so he moved toward a model that decides instead of talks.
Instead of generating text, Jev evaluates program state against a fixed menu of answers and outputs a probability distribution in a single pass. The model has a 32K context window. The pricing is 4 cents per million words of input, with free output. Calls return between 70 and 500 milliseconds.
Developers are already plugging it into semideterministic workflows. Zack Gone reported sorting 20,000 pieces of information about complaints, upsells, and missed follow-ups in 7 minutes, at a total cost of $1. Developer Nakshhatra Sakena built a Jev classifier for an existing tax-document pipeline, reporting 34x less cost and six times faster processing than the prior approach.
In web automation, BrowserUse’s Jev ultrafast reduced page interactions from 1,100 to 101. The runtime went from 9 and 1/2 seconds to 7. The project received 9,000 users in 3 days. In a run across five real coding sessions at a safe confidence setting, the system dropped 0% of the context.
Six days after Jev launched, solo developer Theo Lee released an open-source implementation called OpenJeff, which he later renamed. It runs locally in a browser without a server or account. On the best open model, OpenJeff matched the reference answers about 85% of the time.
Faster Decisions Missed Flaws That Slower Models Caught
Before treating fast probability choices as a clean win, look at the unverified claims and the performance limits. Type Safe advertises zero hallucinations, but its own page admits the number is not measured.
Furthermore, Type Safe’s own accuracy check measures agreement with a panel of frontier models, GPT-6 Astra, and Claude Fable, landing around 68%. As an engineer, agreement with other models is not proof of correctness.
Independent testing also narrowed the speed advantage. In an independent test, Jev ran at about a third of a second against nearly 9 seconds for Claude Fable 5. That is roughly 25 times faster, not the claimed 40 to 200 times, and Jev missed a flaw that the slower model caught. When a system gains raw speed but drops an error that a slower model spots, the tradeoff is real.
Commercial Harnesses Multiply Token Costs for Minimal Gains
The efficiency debate extends beyond model inference into the wrapper software running the agent. A paper from UC Berkeley and LMSYS Arena compared Claude Code, Codex CLI, and an open-source minimal harness called Pi.
The findings reveal a heavy harness tax. Success rates moved only a few points across the harnesses. However, token costs jumped 2 to 5x on proprietary agents because they ship huge first-call context. Because of that overhead, Pi often won on cost-to-performance.
There is also a distinct system-level tradeoff. Codex CLI, written in Rust, uses far less CPU and RAM than TypeScript harnesses like Pi and Claude Code. The tradeoff is plugin and extension flexibility.
Code repositories will wait while an agent pauses to generate reasoning text, but real-time environments will not. Until agents decouple fast reflexes from slow text reasoning, and strip out first-call harness bloat, they will remain too slow for running clocks and too expensive for production pipelines.