The GPU looked great in the cached operator test:
- CPU: 23.98 GB a second
- GPU: 71.37 GB a second
About 3 times faster.
Then the full cold storage query flipped the result:
- CPU path: about 490 milliseconds
- GPU path: about 568 milliseconds
The actual GPU kernel work was only about 14 milliseconds. The card was fast, but it was only doing a tiny part of the total job.
A Fast GPU Kernel Still Lost the Full Query
Cloud-Codes covered a Bamberg paper published on the 12th of September.
The setup was ordinary enough: a desktop graphics card, an 18 core Xeon, plenty of memory, one NVME drive, and a query over 128 million rows and one gigabyte of data on disk.
If you only looked at the cached operator, the GPU looked like the obvious winner. It ran at 71.37 GB a second, while the CPU ran at 23.98 GB a second.
But the full cold storage pipeline told a different story. The CPU path took about 490 milliseconds. The GPU path took about 568 milliseconds.
The kernel itself was only about 14 milliseconds.
So the card computed for 2.5% of the run. The other 97% was spent getting data to it.
That is the useful part. The CPU path stopped at RAM. The GPU path had more steps: SSD to system memory, then PCIe to the card, then the kernel.
For this workload, the GPU was not really competing on pure math. It was competing against the whole data path.
Half the Bytes Helped. More Packing Did Not.
The same paper then changed the data.
It packed 32bit integers into 16 bits because the values fit. Stored data dropped from 1 gigabyte to 512 megabytes.
Now the packed GPU pipeline rose from 1.76 to 3.28. The paper reported 1.86 times, about 60% ahead of the CPU.
So the GPU was not simply losing. Once the data got smaller, the full path improved enough to put it ahead.
But compression was not a free win.
With 12 bit packing on cached data, the result was 212 against 217 without compression.
The paper put the lesson plainly:
“Compression ratio alone is not a sufficient predictor of effective bandwidth.”
Saving bytes helps when moving bytes is the problem. But if decoding those bytes costs too much, the compression ratio stops telling you what the system will actually do.
A 4B Model Can Still Have a 39 GB Cache
The same shape shows up in long context models.
Sparkx 2.5 is a 4B model with just over a million tokens of configured context. That sounds small on the model side and huge on the memory side.
The important bill is not only the model file. It is the KV cache, the model’s working notes for the current conversation.
Spark reduces that cost with its architecture. It has 36 layers total: 27 local layers and 9 global layers. The local window is 512 tokens.
Local layers keep a nearby view. Global layers keep the long history path.
That changes the memory math. The source estimate puts the cache for one million tokens at roughly 39 GB. If all 36 layers kept full history, it would be about 155 GB.
So the architecture cuts the cache to about a quarter.
But 39 GB is still cache alone. It does not include weights, temporary buffers, or the runtime.
This is where download size can mislead people. Full precision GGUF is about 8.2 GB. The 8bit file is around 4.4 GB. The 4 bit file is around 2.6 GB.
Those smaller files shrink the weights. They do not automatically shrink a 16 bit conversation cache.
The scores are split too:
- Swebench Pro: Spark 44.4 versus Quen 3.54B 29.4
- SWEBench verified: Spark 41.6 versus Quen’s 9B 53.1
- GPQA: Spark 67.4 versus Quen’s 9B 77.2
So this is not a clean universal win.
The practical question is not just whether the model loads. It is whether it can read a long repo, remember the right old instruction, make the right change, call tools correctly, and do it fast enough.
DeepSeek Is Cutting the Cache Itself
DeepSeek V4.1 Flash is attacking the same memory pressure from another angle.
CalebWritesCode walked through it, and one number stands out: per token KV cache fell from 389 kilobytes to 890 bytes over the past 3 years.
DeepSeek’s CED, causal encoder decoder, splits the work. Prefill is 8 billion active. Decode is 16 billion active.
Prefill is where the model reads the prompt before answering. If production usage is dominated by input tokens, reducing prefill cost matters.
DeepSeek also uses engram storage for recurring local patterns. The model notes list 196 billion additional parameters during inference.
That sounds enormous, but the source does not describe it as every token activating a normal dense model of that size. It describes it as separate storage for recurring language patterns, injected at layer 1 and layer 15.
CSA 2 goes after repeated KV work across layers. Its hierarchical sparse indexer groups positions into blocks of eight and keeps the top 48 blocks as candidates.
Plainly: DeepSeek is trying not to reread and recache everything the expensive way.
The same practical rule keeps showing up: judge the path, not the part. A fast kernel, a good compression ratio, and a small download file can all be true while the full system is slower or heavier than expected.
The open tests are concrete. Can Spark’s million token setup stay useful when people test real repos, long prompts, memory use, and time to first answer? And in database and analytics work, when does GPU acceleration beat the full data path instead of just the kernel benchmark?