What does local inference mean when generating one token requires reading roughly 17 GB from an SSD?
Running a huge model on consumer hardware is no longer a meaningful threshold by itself. Practical usefulness depends on throughput, memory, storage bandwidth, quantization costs, workload behavior, and reproducible task quality.
Interactive speed changes the equation
Deepseek V4 Flash has 284 billion parameters, with roughly 13 billion active per token. In an early single run, a 2-bit build reached 34 tokens per second on a 128 GB MacBook Pro M5 Max.
That is interactive speed. It suggests that a model with an enormous total parameter count can still be practically useful when sparse activation and aggressive quantization reduce the work required for each token.
But the result is not a complete evaluation. The available evidence does not establish how much quality is lost through 2-bit quantization. Performance with longer contexts and full workloads also remains open.
Quality claims need reproducible, task-specific evidence
Deepseek reports 82.7 on Terminal Bench 2.1, but its harness is forthcoming. None of the nine reported agent scores can currently be reproduced externally.
Artificial Analysis independently scored Deepseek V4 Flash at 50 on its intelligence index. That provides an independent quality signal, but it does not validate the Terminal Bench claim. The intelligence index and terminal task completion measure different things, so one cannot substitute for the other.
SSD streaming trades memory for latency
Waste’s Kimi K3 implementation takes a different approach. Kimi K3 has 2.78 trillion parameters. The implementation keeps roughly 27.28 GB resident and streams selected experts from SSD, allowing the model to run in 29 GB of RAM.
It generates about 0.5 tokens per second while reading roughly 17 GB from storage for every generated token. The model fits, but it is not interactive in the same sense as a system producing 34 tokens per second.
Memory capacity has been traded for storage traffic and latency. The SSD becomes part of the inference system, making its bandwidth central to performance.
Deepseek uses aggressive quantization to reach interactive speed on a 128 GB machine. Kimi K3 uses SSD streaming to fit within 29 GB of RAM, but runs at about 0.5 tokens per second.
Both run locally. That phrase tells us almost nothing about whether either model is useful for a particular workload.
“Runs locally” should begin the evaluation, not end it.