Identical Qwen 3.6 27B weights scored 67.8% in one coding harness and 90% in an engineered stack with retries. That 22.2-point gap shifts attention from the model to its operating environment.
Practical AI performance belongs to the whole system: the agent loop, retries, routing, security boundaries, context, memory, validation, and provenance controls.
The Same Model Can Produce a Different System
On SWE-Bench Verified, Qwen 3.6 27B scored 67.8% with Mini SWE-agent and 77.2% with Qwen’s scaffold. An engineered stack reached 90% without changing the underlying weights.
The 90% result also used 12 graphics cards across two workstations, included retries, and relied partly on a closed-source toolchain. It shows that the harness matters, but does not isolate which intervention produced the gain or establish that an ordinary local setup can reproduce it.
A review of 10 fast-growing GitHub repositories found that six focused on environments around models. The recurring engineering work involved provider switching, security checks, authority boundaries, context, memory, and reproducible validation.
Controls introduce their own trade-offs. DeepSec estimates a 10% to 20% false-positive rate and says large scans may cost thousands or tens of thousands of dollars. A security layer that produces too many false alarms or costs more than the workload can become an operational problem itself.
Routing Cuts Cost, Not Capability for Free
Across LangChain’s 145 multi-turn agent tasks, Nemo Switchyard routed 93% of calls to Lightning and 7% to Claude Opus 4.8. The system reduced total cost by 74%, while accuracy fell from 86 to 80.
The routed result beat Lightning alone by 2.3 points, within run variation of 2.7 points. This test therefore did not establish an accuracy improvement over Lightning by itself.
Routing offers cost control, not a free capability upgrade. Whether the saving is worthwhile depends on the workload and the cost of a wrong answer.
Model Capability Still Sets the Boundary
Artificial Analysis scored Nemotron 3.5 Lightning at 24 on its nine-evaluation intelligence index, compared with 63 for Claude Opus 5. Nvidia reported a SWE Bench Verified score of 51.56 for Lightning, while QN scored 70.12.
Lightning is designed for efficiency. Its hybrid architecture limits the growing key-value cache by using Mamba layers with fixed state. One calculation compared 52 GB of cache for 52 attention layers over a one-million-token conversation with a fixed state of 46 MB for Lightning’s 23 Mamba layers.
Architectural efficiency does not guarantee higher observed speed or stronger capability. Siam Pac measured Lightning at about 72 tokens per second on prose and 84 to 87 on JSON and summaries. QN 3.5 35B reached 78 tokens per second using the same machine, runtime, and prompt. He did not reproduce Nvidia’s four-times speed claim through Alama.
Architecture, training, distillation, quantization, hardware, runtime, and harness choices can all affect the result. Cross-lab benchmark numbers rarely isolate one cause.
Control Includes Custody and Provenance
The operational case for local inference is custody, offline access, and handling private material, not frontier performance. A local system can control where material is processed while trailing stronger hosted models on capability.
Hosted systems are adding artifact-level controls. Anthropic says supported Claude models will embed imperceptible text watermarks, while images and other supported files will receive C2PA provenance metadata. There is no opt-out.
Code remains unresolved. European guidance explicitly exempts source code and several related formats, yet Claude Code remains covered worldwide. Anthropic has not released its detector, technical documentation, or measurements showing whether watermark-driven token selection affects code quality.
Detection would indicate that Claude touched an artifact, not that Claude authored all of it. Editing, translation, or using older models could remove or avoid the mark.
Compare complete systems under the conditions you will actually run them, including retries, hardware, cost, error rates, and control requirements.
Model selection sets the starting point. The usable result comes from the harness, and any claimed gain must include its retries, hardware, closed components, costs, and accuracy trade-offs.