At the final checkpoint of Slop Code Bench, Opus 5.5 at medium failed 258 test cases. 256 were regressions: behavior that had worked earlier and no longer did.
That is the problem with judging an AI coding agent only by its latest change. The new request can work while the project loses something it already had. A useful check has to look backward as well as at the code just added.
New Code Broke Earlier Work
Slop Code Bench tested agents through repeated changes to a project. A strict checkpoint pass required both parts of the job: make the new behavior work and keep earlier tests passing. In the October results, newer models strictly passed only 12.2 to 16.3% of checkpoints.
The final checkpoint shows how much earlier work could be lost. Opus 5.5 at medium failed 258 test cases, including 256 regressions. GPT6 Astra at extra high failed 1,435, including 1,178 regressions. Those are failures at a particular checkpoint in this benchmark, not a measure of how either model performs on every coding task.
The benchmark also tested whether prompting could keep the project under control. One prompt reduced initial verbosity by roughly 28 to 36%. That changed how much the agent said at first, but neither prompt stopped the later decline.
Both prompts added about 12% to the cost of each checkpoint, while correctness slipped 2.3 points. In this test, getting shorter initial output did not preserve earlier behavior as the work continued. The check that mattered was whether the accumulated project still passed.
There is a simple distinction here. A new feature passing its own test says something about that feature. It does not tell you whether an earlier feature survived the edit. Slop Code Bench made agents meet both conditions at once, and the strict pass rates show how hard that was in its repeated-change setting.
The Old Service Gave the Rewrite Something to Match
A separate team had Claude rewrite the Claude messages API from Python to Rust. Before the rewrite, the team planned to send the same real traffic to both versions and compare their responses. A speaker reports 40 to 50% less CPU use.
The CPU figure is the speed result. The planned response comparison addresses a different part of the rewrite: whether the new service behaves like the old one when it receives the same traffic. The old service’s responses gave the team something concrete to check against.
That is related to the benchmark’s earlier tests. In one case, an agent must add behavior without breaking what previously passed. In the other, a rewrite has an existing service whose responses can be compared with the new version. Both give the new code a reference beyond “the latest change runs.”
The scope matters. The account reports lower CPU use and describes a plan to compare responses. It does not give a result for that comparison. The CPU number alone cannot tell us whether every response matched.
A Passing Chrome Check Missed Safari
Checks can miss behavior that users actually reach. One team checked video playback in Chrome, but that check missed a playback problem in Safari. The team later added Playwright WebKit and a real Mac to its private check.
The Chrome check covered Chrome. The Safari failure shows what happened outside that coverage: a check passed while the viewing experience was still broken. Adding WebKit and a real Mac brought that missed experience into the team’s check.
This is the other half of the coding problem. Earlier tests can catch regressions only in behavior they cover. Comparing an old and new service can check responses only where the comparison is made. The Safari example is a reminder to examine the check itself when the user’s experience disagrees with it.