D-Fly ran 1.98 to 2.40 times faster than ordinary decoding across the paper’s tests.
The more useful finding from AngelSpec is less convenient: no single speculative-decoding design worked best across real workloads. The best choice changed with the work, the training, and even how requests were scheduled.
No Drafter Wins Every Workload
Normal AI generation produces one token at a time. For each token, the chip loads the large model’s weights and runs them again.
Speculative decoding changes that loop. A smaller draft model guesses several tokens, then the large model checks them together. It keeps the valid prefix, corrects the first disagreement, and repeats.
The surviving tokens follow exactly the distribution the large model would have produced alone. The claimed benefit is faster generation without a quality trade-off.
Tencent’s AngelSpec puts six speculative-decoding methods into one training pipeline, letting serving teams compare them against the same target model and their own traffic.
Different workloads favored different specialists. Conversation is open-ended, so long guesses are more likely to fail. Code and mathematics are more predictable, so guessing an entire block can work better.
Tencent trained multi-token prediction for conversation and block diffusion for code and mathematics.
Honeycomb 3, for example, adds a 3.8 billion parameter prediction layer inside a 295 billion parameter model.
D-Fly Was Fast, but Training Mattered Too
Accepted length measures how many draft tokens survive each verification round. Every accepted token after the first avoids another trip to model memory.
On Honey 13, accepted lengths were:
- MTP: 3.00
- D-Flash: 3.69
- D-Fly: 4.79
On Twin 3:
- D-Spark: 5.32
- D-Fly: 5.41
The speaker calls that close enough to a tie.
Across tests with four to 64 concurrent users, the paper reported D-Fly running 1.98 to 2.40 times faster than ordinary decoding end to end.
But architecture was only part of the result. With the architecture fixed, Montana Piatesi changed the training approach, and mean acceptance moved from 52.8% to 66.4%.
The D-Fly ablation showed the same pattern:
- A backbone change moved accepted length from 3.77 to 4.40
- An autoregressive head took it to 4.60
- Code and mathematics training data pushed it to 4.75
The drafter matters. So does the data used to train it.
Lower Acceptance Still Produced More Throughput
AngelSpec also includes Dcut, which distributes verification work across a group of requests using drafter confidence and a hardware cost model.
On live Honey 1 traffic with 64 concurrent requests, Dcut delivered 15.7% more throughput than D-Fly alone.
At the same time, mean accepted length fell:
2.50 → 2.46
A slightly worse isolated metric still produced more total throughput. One number cannot describe the whole serving system.
The reported speed figures of 4.6 times, 2.86 times, and 1.98 to 2.40 times came from different models, hardware, and loads. They are not one clean comparison.
The Easier Tool Drew More Attention
AngelSpec may be the more useful comparison toolkit, but Dflash2 received far more attention.
Dflash2 reached 99 points and 18 comments on Hacker News. One model file received about 70,000 downloads in under a week.
AngelSpec had:
- 219 stars
- 19 forks
- One watcher
- 1,773 downloads across seven model files over 30 days
It had not been submitted to Hacker News.
Usability may be part of the gap. Dflash2 has MLX support for Apple hardware, OpenAI-compatible servers, an MIT license, and more than 130,000 pulls. The speaker says it works and prefers it for faster local generation.
AngelSpec asks teams to choose a target model, generate hidden states, train a drafter, and serve the result.
Its repository also looked unfinished. It had three commits, with the latest on July 31, three open issues, and an unmerged pull request with no comment for more than three weeks.
Reported implementation problems include:
- A refill task that can fail silently
- Cleanup code that can delete the best checkpoint while its pointer still names it
- A loss path that breaks the D-Spark head
AngelSpec gives teams a way to test six methods against their actual model and traffic. But they still have to train and serve the drafter, and the open implementation problems make that harder.
A 90% Cache Discount Does Not Make Every Token Useful
API builders repeatedly send system prompts, tool definitions, and reference documents. Prompt caching stores that stable context so it does not need to be processed at the standard price every time.
According to @NateBJones, Opus cache hits cost 50 cents per million tokens, compared with $5 per million normally.
That is a 90% discount on repeated content. Cheap context, however, is not automatically useful context.
The @Cloud-Codes brief says an Anthropic fix on the 17th of August reduced the context needed to load Claude Code’s built-in Claude API skill:
More than 200,000 tokens → about 25,000 tokens
The practical response is to audit what plugins and skills load, remove unnecessary material, and cache the stable content that remains useful.
Agents Still Need a Clear Finish Line
Efficiency is not only about tokens and throughput.
@NateBJones says agents performed sophisticated, relentless work, but none of it was wanted. His point is that a team unable to define what “done” means before installing an agent may buy process instead of value.
A separate @aicoachjohn brief describes a Toyota setup involving deep agents, Langraph, Langsmith, GitHub, and a platform called “50 plus action.”
The speaker focuses on six engineers becoming one.
“AI didn’t remove the engineer, it removed the headcount around the engineer.”
The stated response is for workers to stop acting as task workers and learn to own the system.
The agent stories and inference results point to the same operational problem: more activity is not automatically the intended outcome. Someone still has to choose the workload, define success, and own the system.
Can independent teams reproduce AngelSpec’s results after its open implementation issues are fixed? In real deployments, will specialized drafters remain better than one general-purpose design?