A 75% one-attempt success rate falls to a 42% completion probability when a workflow requires three consecutive successes. A plausible result once is not the same as a dependable workflow.
That gap is the verification tax. AI can reduce the effort needed to produce code, notes, or tool calls, but some of that work moves into evaluation, review, and error analysis. Engineers still have to choose the system architecture, define success, and prove that repeated execution works.
Code mode wins only when the workload earns it
A PwC benchmark compared programmatic tool calling with JSON function calls.
Code mode showed clear advantages for demanding workloads. It matched or beat JSON for 11 models, while 13 models matched or beat JSON on parallel fan-out. At chain lengths of 12 or more, code mode held an 18.8-point advantage. Chaining took roughly half the wall time for 13 models.
But JSON retained the higher per-category macro average across all models: 78.6 compared with 77.0. It also remains cheaper and simpler for small, shallow tool sets.
Model reliability complicates the architecture decision. GPT-4o, GPT-4.1, and GPT-5.4 Mini produced malformed newlines in Python, causing their scripts to fail before any tool ran. After removing those three models, the reported code-mode gain at 128 schemas fell from 5.5 points to 0.6 points.
Programmatic calling becomes attractive with more than about 26 tools, deeper chains, or large fan-outs. That does not make it a universal replacement for JSON. The workload must have enough breadth, depth, or parallelism to justify the added complexity, and the selected model must execute it reliably.
The benchmark stops before the hardest part
The PwC benchmark used echo-return stubs. It tested argument serialization, not complete workflows where one action’s result determines the next action.
That limitation belongs beside the strong long-chain result. Better serialization across many calls does not establish dependable end-to-end execution when later decisions depend on earlier outputs.
Each dependency creates another opportunity for failure. A convincing isolated output can hide that compounding risk.
Engineering judgment becomes more consequential
Andrew Ng’s four-part skills map places building and deploying AI applications, software fundamentals, coding agents, and shaping the build ahead of prompt engineering. Its central engineering claim is practical: unpredictable model output requires disciplined evaluations and error analysis.
Software fundamentals remain necessary for judging cost, scalability, reliability, security, and privacy. Faster generation does not settle those decisions. It makes identifying failure modes, defining acceptable behavior, and testing complete sequences more important.
Nate Jones offers lightweight controls for that work. Choose the simplest system shape that fits, then maintain three files:
project.mdfor requirementsdecisions.mdfor consequential choicesscenarios.mdfor testing real situations after generation
These files make the intended behavior and chosen trade-offs explicit. They also shift attention from whether the generator produced something plausible to whether the finished system behaves correctly in actual use.
Choose the simplest architecture that fits the workload, then test the full sequence repeatedly, not just one convincing output.
Models may handle more implementation, but engineers must still choose the system shape, record consequential decisions, and prove that repeated execution works.