Quinn 3.8 27B reportedly needs 62.5 MB of cache at an 8,000-token context. A generic formula predicts 250 MB because it assumes all 64 layers retain a growing cache, while only 16 use full attention.
The calculation may be valid for its assumptions, but those assumptions describe the wrong architecture.
Gemma 4 12B shows the same problem. It uses bounded sliding attention in 40 of its 48 layers. Its reported cache requirement at the same context is 832 MB rather than 3 GB.
Architecture therefore belongs in memory planning. A generic calculator can produce a precise number that does not describe the model being deployed.
Less memory can cost accuracy and speed
Cache quantization reduces memory further, but the most aggressive setting is not automatically the best.
Q8 reportedly saved 47%, with at most four answer changes across 500 deterministic reasoning questions. Q4 saved 72%, but accuracy for the older Quen 2.5 model fell from 92% to 24%. At 64,000 tokens, Q4 was also 35% slower.
The trade-off is not simply memory versus accuracy. More aggressive quantization can hurt speed as well. Based on these results, Q8 is the more defensible default until the specific model and workload have been tested.
A faster benchmark is not the whole cost of an engine
Free Token’s routing-aware cache reportedly reduced expert read misses from 62% with a static split to 16%. Its authors reported 14.9 tokens per second for GLM 5.2, compared with 7.3 using llama.cpp. Reported worst-turn latency was below 44 seconds, versus 232 seconds for llama.cpp.
The technique merits testing, particularly on the benchmarked configuration. The evidence does not yet support a general recommendation to switch engines.
These were first-party benchmarks. Free Token was beta software limited to Nvidia CUDA on Linux, with two contributors and 30 public commits. Llama.cpp supported 17 hardware backends and had 446 contributors.
An inference engine costs more than its best benchmark. Hardware coverage, software maturity, contributor depth, and operational fit all affect adoption. Independent reproduction of the latency gains would strengthen the case. So would adoption of routing-aware expert caching by established engines before Free Token expands its hardware support.
Never choose a local AI configuration from a generic estimate or headline benchmark. Test the specific model, workload, hardware, accuracy, and latency together.
For now, Q8 is the more defensible default pending testing. Free Token’s routing approach is worth watching, but its current benchmarks are not a general reason to switch engines.