A benchmark is only honest if you keep running it. Ours had started to rot in a way that is easy to miss and embarrassing to admit: the independent judge panel was pinned to model names that no longer resolve against the API.

This post is the cleanup. We fixed the panel, re-ran the full public catalog on current generally-available models, fixed a cost-accounting bug we found on the way, and — the part that actually matters — connected the benchmark's autonomy score to the autonomy controls OrgX shipped this month.

The bug: judging with models that no longer exist

The last published judged run used a panel of gpt-5.4-nano, gpt-5.4-mini, and gpt-5.4. When that bundle was produced, those were the current models. By the time you re-run months later, "the public judge panel" silently means "three model IDs that 404."

The default judge preset now pins to current GA models, chosen to keep the panel diverse without overpaying for it:

The generator stayed gpt-5-nano on purpose. We wanted to change one thing — the judges — so the re-run is comparable to the last one instead of being a new experiment.

The second bug: judge cost was reading as zero

While wiring up the new panel we found the cost table was missing entries for gpt-5-mini and gpt-5.1. The cost estimator returns null for any model it does not know, which quietly summed to zero for two of the three judges. The previously reported judging cost was undercounting, because most of the panel was free in the spreadsheet and not in reality.

That is exactly the kind of error a transparency benchmark should catch and say out loud. The pricing table now carries the current gpt-5.x models, so the cost numbers below are the real spend.

The run

We ran the full 15-task catalog, 7 domains, 2 repeats per task, and 3 independent judges per generated artifact.

Headline metrics, recomputed from the raw scorecard and cross-checked by the bundle validator:

A note on that last row: the previous judged run flagged 45 of 45 artifacts for human review. The corrected panel flags 23 of 30. The review flag is a sensitive triage signal — it fires on any material judge disagreement, any judge recommending review, or any judge failure — so a drop from "everything" to "most things" is the panel becoming less of a blunt instrument, not the work getting suddenly trustworthy. It still needs human calibration before it backs a public claim.

You can inspect every file behind these numbers:

• summary.json: headline metrics and costs
• examples.json: the generated artifacts
• judgments.json: per-judge criterion scores and disagreement
• scorecard.csv: task-level scoring
• tasks.json: prompts, rubrics, and human baselines
• metadata.json: runner provenance

What it cost

The full run cost 52.99¢ — about half a dollar for 30 generated artifacts and 90 judge calls.

The takeaway is unchanged from the last run, just measured correctly this time: the cost driver is the high-reasoning gpt-5.1 judge, not the generator. Cheap models can keep doing the work while a stronger model judges it, and the whole panel still costs under two cents per task.

The part that matters: autonomy is now scored against real gates

The single most important number this benchmark reports is autonomous_completion_rate. It is only credible if "autonomous" means the same thing in the benchmark and in the product. This month OrgX shipped a real autonomy control plane, so we tightened the definition to match it.

What shipped in OrgX:

• Gated-autonomy modes — every initiative runs in manual, gated, or

autopilot. manual waits at every step; gated (the default) auto-continues low-risk work and holds key artifacts for review; autopilot ships low and key artifacts automatically. A high-risk safety floor — external sends, spend, deploys, merges, deletes — never auto-approves in any mode.

• Eval-gated artifact auto-approval — when an initiative's mode permits it,

a run's final deliverable is judged by a precision LLM judge and auto-approved only if it clears a ship-ready bar. Anything short of a clear pass stays in review and keeps iterating.

• Auto-approval on a real eval pass — for artifact types with a bound eval,

a recorded passed status is the criteria; the artifact auto-approves with no judge at all. It is the most conservative gate in the set.

• An OrgX-owned autonomy decision endpoint — worker runtimes ask the OrgX

backend whether they may proceed before parking a step, so the same trust boundary is enforced across Agent, API, CLI, and sandbox runtimes instead of being re-implemented per surface. Any error fails closed.

The benchmark now defines a clean autonomous completion as a run where every required artifact reached an approved state either by non-human self-heal or by eval-gated auto-approval, under a mode that permits it. If a human approval was required — including anything the precision judge routed to a human or the high-risk floor caught — the run is preserved and scored on quality, but it does not count as autonomous.

This is deliberately stricter than "the agent produced an artifact." Producing something a human still had to approve is assistance, not autonomy. The full restatement, including the new per-artifact fields (autonomy_mode, approval_path, precision_judge_outcome, safety_floor_triggered), is in the methodology amendment in the public repo.

Why this matters for Initiative Worlds

The catalog you can run today is the transparency surface, not the frontier suite. The headline-eligible work is the Initiative Worlds split: seeded OrgX workspaces with hidden evaluator truth, deterministic validators, and approval boundaries. Both runnable preview worlds — Activation Sprint and the Cross-Functional Launch Gate — pass their deterministic validators with zero hard-fail flags, including the requires_human approval and trust-preservation checks.

Aligning the catalog's autonomy definition with the product's autonomy gates means both halves of the benchmark now reward the same behavior the platform is built to produce: do everything that is safe to do automatically, and stop at the boundary that genuinely needs a human. An agent that performs an action behind a requires_human boundary without approval fails trust outright, regardless of how good the artifact is. An agent that correctly holds for approval is not penalized — that is the gate working.

What we are still not claiming

• The precision-judge threshold is not calibrated. Like the judge-disagreement

threshold, it needs human adjudication before it backs a strong public claim.

• The human baselines are still expert estimates with small samples. They are

labeled as such in tasks.json, and a +10-point quality delta over an estimate is a signal, not a verdict.

• A high autonomous_completion_rate on the public catalog is a completion

signal, not proof of organizational competence. That is what the private holdout worlds and timed expert baselines are for.

Where this leaves the benchmark

The benchmark runs again — on models that exist, with costs that are real, and with an autonomy definition that tracks the product instead of drifting from it.

The next target is the first platform-backed run that reports approval_path for every artifact — so the autonomy rate is not just "the agent finished," but "the agent finished, and here is exactly which gates it cleared on its own and which it correctly held for a human."