Benchmarks

SGT Fleet benchmarks

SGT internal measurement: 80% fewer frontier tokens with the frontier model forced into orchestration mode. JEV's separate 50% is a cost cut on eligible remaining work; 90% is conditional arithmetic only.

Last updated 27 September 2026

How much does SGT Fleet reduce frontier-model usage?

SGT reports an internal measurement of 80% fewer frontier tokens when the frontier model is forced into orchestration mode. This is an SGT-reported internal measurement, not an independent audit. The detailed counts and method behind it are not yet public.

In the Blender scene study below, SGT Fleet on DeepSeek Flash 4.1 reached its final build with 0.92 million total tokens against 1.90 million for Claude Sonnet 5 working alone. Flash 4.1 alone, without SGT Fleet, used even fewer tokens (0.72 million) but needed eight attempts and scored lower.

JEV is a separate result: in one repeatable routine workflow it cut frontier orchestration cost by 50% on the eligible remaining work, with the frontier model still creating templates and reviewing results. That is a cost measure, not a token count, and it is separate from the 80%. The 100 to 20 to 10 path, sometimes shown as 90%, is conditional sequential arithmetic on top of the 80% figure, not a combined result measured end to end. No guaranteed bill reduction is claimed. Claims status.

Does SGT Fleet improve output quality?

It depends on the worker model. In the Blender scene study, SGT Fleet raised DeepSeek Flash 4.1 to within 0.12 points of the reference build. It made no measurable difference with Claude Sonnet 5, and with GPT-5.6 Luna it made results worse: Luna alone passed, while Luna with SGT Fleet never rendered a passing frame. SGT Fleet's learned skills were built from Flash 4.1's failure modes, so they help Flash most.

Blender scene study: German spring village home (16 September 2026)

Ten ways of producing the same Blender 5.2 scene from an identical brief. Every arm ran the same headless render and the same automatic gate (script ran, four 1600×1100 views, not blank, exposure in range, a calibrated vision read of every view). The orchestrator added repair rounds until an arm came within 0.5 points of the reference or reached attempt 8. Two judge families, Claude Opus and Codex, scored all finals blind in one sitting under sealed label mappings, giving eight scores per arm on a 1–10 scale.

Cost of reaching the final build

What it showed:

• Splitting the scene into separate bounded modules (house, site and camera driver), and repairing one module per round, took Flash 4.1 with SGT Fleet to 7.66 in five rounds and 45 minutes.
• SGT Fleet is not model-agnostic yet: it helped on Flash 4.1, made no measurable difference on Sonnet 5 and hurt on Luna.
• Orchestration errors cost as much as model errors: two faulty orchestrator instructions broke three arms during the study.

Does SGT Fleet work with Codex and OpenAI models?

Yes. SGT Fleet has been paired with OpenAI models in the studies on this page. In a Blender cinematic study with judging accepted on 3 September 2026, GPT-5.6 Sol with a trained SGT specialist scored 8.21 (SD 0.16) against 7.54 (SD 0.15) for Sol alone: 8.9% higher, and higher in all five paired cells (four wins, one tie within the 0.5 margin). Ten verdicts were given across five order-cancelled cells on one brief. Both builds passed all 22 inspector checks; the SGT render's colour fingerprint stayed outside the target band. Two other OpenAI models, used without SGT, did not finish.

Is SGT Fleet faster?

In a small Python-utility pilot on 5 September 2026, the same DeepSeek Flash model ran through two harnesses. SGT Fleet with its critic passed 4 of 5 tasks strictly and 62 of 63 behavioural checks, in 33.15 seconds on average. Claude Code's harness with the same model matched 4 of 5 and 62 of 63, in 55.40 seconds. It was a small pilot with uncontrolled concurrency, so treat it as a routing signal, not a model ranking. In the Blender study, the SGT component arm finished in 45 minutes; the other arms took between 1 h 32 min and 5 h 53 min.

How we measure

• Identical briefs and delivery contracts for every arm, checked by hash.
• A deterministic gate on the delivered files before any human or model judgement.
• Blind judging with sealed label mappings, using judges from more than one model family where stated.
• Token counts for the published studies on this page come from the bridge ledger that records every model call; scores are means over the stated number of ratings. That method applies to the published studies, not to the 80% first-party measurement, whose counts and method are not yet public.
• Older studies on this page carry their date and sample size. The first-party 80% measurement's detailed counts and method are not yet public. The 90% figure is conditional sequential arithmetic, not a measured combined study, and is not presented as a single measured number.

Limits of these results

• Each study covers one brief or one scene. Results on another scene, date or model version can differ.
• Scores vary between runs; single cells have flipped when re-run, and the two judge families disagreed on parts of the ranking.
• In the Blender study, the repair cap was raised from 3 to 7 during the run, and SGT's rules were extended while it ran.
• A historical visual study (1 September 2026) scored trained SGT 7.44, stock Sonnet 6.72 and a frontier hand build 6.38. Those were five ratings of fixed renders by one judge, not five builds, and the raw data is not retained.
• These are visual and task scores, not business returns. Paying for SGT Fleet does not by itself lower what you pay your model provider.

Benchmark claims status

The 80% is SGT's own first-party measurement, taken with the frontier model forced into orchestration mode — it is measured, not a target. The raw counts and method are not yet published; publishing them will document the result, not change its status. Results vary by task and workload. Benchmark boundaries and disclosures: disclaimer. Install and use: docs. Questions: support.