Most backend workflows do not require an AI model to write prose. If the job is sorting a record, answering yes or no, or rating an input on a scale, routing it through a conversational chat endpoint is slow and expensive.
Reading Raw Logits Cuts Input Costs to 4 Cents
Hosted Jev, created by ex-ChatGPT team member Diogo Almeida at Type-safe, takes a different path. Because Jev never writes a paragraph, it is priced at 4 cents for a million words of input and answers back in well under a second. Diogo Almeida, who was on the original team that built ChatGPT, created Jev after concluding chat models did not deliver, designing a system that decides instead of talks. Instead of generating text, the system reads the raw logit for each allowed option directly from the model in a single forward pass, selecting the highest score.
Reusing the key-value cache across multiple questions about the same document unlocks massive speedups. On an open build, scoring decisions from scratch processes a couple per second, whereas reusing notes across questions with shared context reaches about 20 per second on the same card.
Semeth applies raw logit scoring directly to an existing 4 billion Qwen model with zero fine-tuning. On a shared benchmark of about 100 aligned questions, it scores about 81% against Jeff’s 88, running more than five times faster than having the same model output JSON.
Local Models Route Fast on Modest Hardware but Slip on Unseen Tasks
For hospitals, banks, and enterprises under strict privacy agreements, sending internal state offsite prevents automation entirely. Several open-source alternatives now run these workflows locally. Kev is built on Alibaba’s open Qwen 3.5 weights and is offered in three sizes: 0.8 billion, 4 billion, and 9 billion parameters. Palmer notes that the 4 billion and 9 billion models both fit on a 32 gigabyte Mac, with the 4 billion serving as a practical sweet spot for mid-range cards.
On trained tasks, the 4 billion and 9 billion models score about 87% right on Jeff’s heels at 88. On unseen tasks, the 9 billion holds up around 85, the 4 billion drops to the high 70s, and the 0.8 billion falls to 65.
Designed for constrained hardware, Vaughn uses ModernBERT, an encoder pre-trained on 2 trillion tokens that judges text rather than generating continuations. The model contains 395 million parameters, takes about a gigabyte and a half on disk, runs on CPUs, older laptops, or phone-class chips without a GPU, and returns decisions in under 15 milliseconds. On a clean routing task with 20 buckets, Vaughn scores about 83%, but drops to 71 across a suite of 49 tasks.
Sandbox Escapes and False Alerts Expose Execution Risks
For data engineers, logit scoring is ideal for high-volume, low-stakes automation like ticket triage or browser element selection. For high-stakes workflows like loan approvals, human review remains necessary. Giving models execution authority over systems requires dependable sandboxing and strict output verification.
Researchers escaped the OpenAI Codex sandbox and ran commands on the host. In addition, users reported that WSL with the latest Codex extension and CLI 0.155 could not run sandboxed commands.
Verification failures carry even higher stakes. An AI-assisted intelligence report issued a warning regarding nuclear weapons and immediate military action. An analyst and the report circulated through military channels without proper verification. The incident showed the critical risks when automated outputs bypass rigorous verification.
Node FFI Delivers Native Speed with 37 Nanoseconds of Overhead
When a Node backend slows down under load, teams often consider a full rewrite in Go or Rust. That is usually unnecessary. Profiling with --cpu-prof typically reveals that a single expensive function causes queues to back up and tail latency to spike.
Photo resizing illustrates the bottleneck. Resampling requires a tight loop running millions of times, which runs slowly in pure JavaScript. On its own benchmark, Sharp (a native image library for Node that wraps the C library libvips) resizes this type of photo roughly 26 times faster than pure JavaScript.
Node FFI lets JavaScript call shared libraries (.so, .dylib, or .dll) directly without writing C++ wrappers or running node-gyp. Node FFI arrived in the spring behind a flag in Node 26.1 and is enabled by default in the newest release, where it remains marked experimental. A public benchmark measured FFI overhead near 37 nanoseconds per call compared to roughly 35 nanoseconds for a compiled C++ add-on. For operations passing large buffers, this translation tax is negligible.
Native calls carry important constraints. Native FFI calls execute synchronously on the main event loop thread. Offloading calls to a reusable worker pool keeps the main thread responsive. Furthermore, Node’s documentation warns that the API is unsafe: providing incorrect type signatures or invalid pointers does not produce a clean JavaScript exception, but crashes the entire process.
The practical move is to encapsulate FFI calls inside a single well-tested wrapper, measure with the built-in node --bench tool, and reserve full rewrites for when native compute is scattered across the entire codebase. The central question to watch as these systems mature is whether open-source local decision models can match hosted logit calibration without requiring task-specific fine-tuning. Logit scoring is ideal for high-volume, low-stakes automation like ticket triage, but high-stakes workflows still require human review, and autonomous agents still require strict sandboxing.