The same model solved 33.2% of coding tasks in Claude Code and 62.1% in Mini Sweep Agent. Hugging Face changed the app running the model while the model weights stayed the same. That is a wide spread for a result we might casually call a model score.
The Same Model Scored Differently Across Four Apps
Hugging Face tested Liquid AI’s LFM 2.5, a 2.6 billion parameter model, on 250 tasks it had not seen during training. Here is how often it solved them in each app:
- Claude Code: 33.2%
- OpenCode: 33.6%
- Codex: 40%
- Mini Sweep Agent: 62.1%
Claude Code was the app in this test, not the model. The model weights stayed the same across these runs, yet the measured result changed with the app. If you see one of these percentages on its own, it leaves out something essential about how the test was run.
The scores tell us that the app was part of the measured result. They do not tell us which part of an app produced the gap. Hugging Face did not isolate that cause, so the comparison supports a narrow point: on these 250 tasks, this model got different scores when different coding apps ran it.
That is enough to make “model score” a less simple label here. The number belongs to a test of a model working through an app. Treating it as a property of the weights alone would miss what changed between the runs.
Training Through the Apps Improved First-Try Results
Hugging Face then trained the model through the apps. A proxy recorded the exact tokens the model generated and joined its calls into complete task attempts, including retries. Correct answers earned rewards, with a bonus of up to 0.1 for using fewer tool calls.
After training across four apps, average first-try success rose from 42.2% to 54.2%. On tasks that both versions solved, the trained model used 31% fewer tool calls. The qualification matters: that tool-call figure covers tasks both versions finished, rather than every task in the test.
These are two related results, but they answer different questions. The initial four-app comparison held the model weights steady and changed the app. The training result shows what happened after Hugging Face trained the model using complete attempts made through those apps. Keeping those steps separate makes the evidence easier to read.
The retries were part of what the proxy recorded, even though the reported success change was for first tries. The reward also took tool use into account. So the training result includes both a higher average first-try success rate and fewer tool calls on the tasks both versions could solve.
Broader Training Did Not Give OpenCode Its Best Score
The four-app training run did not produce the best OpenCode result. Training only in OpenCode brought its score to about 58%. Training across four apps brought it to 50%.
It would be tempting to turn that into a rule about training in one app versus several. This comparison cannot carry that rule. The four-app run used more data, and Hugging Face ran each setup once. Those differences leave the OpenCode result as an observed comparison, not a settled explanation of why one setup scored higher.
The same limit applies to the original spread. We can see the scores, from Claude Code’s 33.2% to Mini Sweep Agent’s 62.1%, without assigning the gap to a particular app feature. Hugging Face did not test which part of an app caused it.
For this test, the practical reading is straightforward: a coding score reflects the app running the model as well as the model. The original gap is clear. Its source within the apps remains open.