In a reported RTX 3090 test, a merged llama.cpp change raised decoding from 22.97 to 42.45 tokens per second. The same test had two less appealing results: prompt reading fell from 1,315 to 665, and video memory use rose by roughly 2.5 GB.
The decoding gain is real in that reported test. Whether it helps a particular workload depends on more than the generation speed alone.
Faster Decoding, Slower Prompt Reading
The llama.cpp change uses the model’s built-in MTP head to draft tokens. In the reported test, that helped the model generate text faster after it had read the prompt. Reading the prompt itself got slower, while the test also needed more video memory.
Those measurements belong together. A result that says only 42.45 tokens per second leaves out the fall in prompt reading speed and the extra memory used to get there. The change has a clear benefit for decoding in this test, with costs elsewhere in the same run.
Other local AI efforts use different ways to pursue speed. Fusing operations cuts overhead between GPU tasks. Speculative decoding has a small model guess tokens for a larger one to check. Expert offloading moves model parts between GPU memory, system RAM, and storage.
Those methods explain what an implementation is trying to do. They do not, by themselves, tell us how fast the whole system ran or what it was compared against.
The Baseline Changes What a Speed Claim Means
The Strata Pascal fork reports 24.53 tokens per second against 7.88 for llama.cpp on a Qwen 3.6 model at an 8K prompt. That looks like a straightforward comparison until you try to inspect the llama.cpp setup: the fork does not disclose the offload flags used for its baseline.
Lithos Metals lists 168 tokens per second at short context and 127 at 32K on an M5 Max. Lithos calls these target compute projections and says the timing scopes differ. The two figures therefore do not establish a direct end-to-end speedup.
Polymath’s headline comparison pairs 57 with 26 for llama.cpp. The 26 does not appear in its published readme, backends document, or release notes. Without that number in the published documents, a reader cannot check the baseline from those materials.
These are different limits. One comparison leaves out settings used for the baseline. Another gives projections measured over different scopes. A third uses a headline baseline that is absent from the published documents. Keeping those distinctions attached to the numbers is more useful than treating every tokens-per-second figure as the same kind of result.
The iPhone’s 3 GB Claim Sits Beside 16.6 GB of Files
Open BMB says its 35 billion parameter mini CPM V 4.7 model runs on an iPhone in 3 GB of memory. The files stored on the phone total about 16.6 GB. Open BMB validated the app on an iPhone 17 Pro Max with 12 GB of RAM.
The app does not have to use every expert for every token. It keeps experts in 40 shard files and loads them from the SSD as needed. A third-party reading of the configuration says each layer selects eight of 256 experts.
The app also tries to load the next layer’s experts early. A wrong guess costs time, but the router still decides the output. That loading behavior helps explain how a working-memory claim can sit beside much larger stored files.
Open BMB reports more than 13 tokens per second on the validated phone, though that speed has not been independently reproduced. Its documentation does not define what the 3 GB counts or say how often those early loading guesses are right. Open BMB recommends 12 GB of RAM and says the app runs considerably slower with less.
The memory figure and the file size can both be part of the account. Until the 3 GB figure is defined, it is hard to say exactly what that headline covers. The RTX 3090 test gives a fuller picture of its own speed gain: faster decoding, slower prompt reading, and roughly 2.5 GB more video memory use.