A benchmark should measure its own errors
On July 8, 2026, OpenAI published an audit estimating that roughly 30% of SWE-Bench Pro's public tasks were broken. The defects were not exotic: tests that demanded one implementation, prompts that omitted tested requirements, tests that accepted incomplete work, and prompts that pointed in the wrong direction.
That finding changes the burden of proof. A benchmark cannot earn trust merely by hiding tests, adding more tasks, or running stronger models. It has to measure the probability that the benchmark itself is wrong.
OrgX-Bench V1.2 makes that a release gate.
What OrgX-Bench measures
OrgX-Bench does not ask whether an agent can produce a plausible document. It asks whether a system can move an organization from one valid state to another while preserving trust.
A qualified success requires all of the following:
mission complete
AND final organizational state valid
AND work product accepted in downstream use
AND evidence and provenance valid
AND approvals and policies respected
AND no critical trust violationThe intended frontier headline is the Trustworthy Initiative Horizon: the longest human-equivalent initiative duration at which a system sustains 80% qualified success under a fixed stress profile.
We publish the underlying dimensions separately. The horizon is not permission to hide reliability, cost, trust violations, or human rework inside one weighted score.
The world-quality court
Every private generator must now carry a quality dossier before it can contribute to a headline:
- five independent reviewers
- at least two independently valid solutions
- at least three partial, shortcut, invalid, or policy-violating solutions
- measured false-accept and false-reject rates
- an ambiguity rate and reviewer-agreement result
- counterfactual twins
- metamorphic tests
- delayed-consequence tests
The validator has to accept the valid equivalence class, not only the solution its author imagined. It also has to reject plausible shortcuts for the intended reason.
The maximum false-accept rate is 2%. The maximum false-reject rate is 2%. Severe known defects are not averaged away; they block the world.
Counterfactual twins
Memorized patterns often survive ordinary paraphrases. Counterfactual twins are stricter.
Two instances can present nearly the same incident, renewal, migration, or approval narrative while changing one authoritative fact. The correct action must change with that fact. An agent that repeats a familiar remediation instead of tracing the causal state fails the pair.
This tests whether the system responds to organizational truth rather than benchmark familiarity.
Private is not the same as uncontaminated
Private worlds use just-in-time seeds, a sealed evaluator vault, signed access events, provider-retention controls, planted canaries, and adaptive leakage probes.
A strong leak signal burns that world for the affected release. The world remains visible in the evidence ledger, but its score cannot appear in the headline.
This is intentionally expensive. Quietly assuming that a private file stayed private is cheaper and scientifically weaker.
Eight runs are not enough by definition
V1.2 keeps eight episodes per cell as a minimum. It is no longer the stopping rule.
Paired-seed sampling continues until the confidence interval reaches the preregistered width or the budget cap is reached. If uncertainty intervals materially overlap, the publication suppresses the rank instead of manufacturing a winner.
Corrections are part of the benchmark
The public correction ledger is append-only. Severe and critical reports block affected releases while open. A resolved severe defect requires a public explanation and score recomputation or withdrawal.
Task authors cannot be the sole approvers of their own graders. Outside reviewers are asked to break the benchmark, not endorse it.
What is implemented now
The public repository now contains executable schemas and validators for:
- world-quality audits
- contamination audits
- statistical-precision reports
- the correction ledger
- strict release integration for all four
- permanent negative controls proving the validators fire
Benchmark Lab also distinguishes mechanism evidence from headline eligibility and can display each missing quality gate.
What is not complete
The current V1.2 release remains preflight. It does not yet have completed private world dossiers, the 60 timed expert sessions, a precision-qualified frontier sweep, independent replication, or outside reproduction.
That is the point of publishing the contract first. The methodology is now public before the result exists, and the result cannot become a headline merely because the execution finished.
The implementation and evidence state are available in the public benchmark repository. The complete methodology is at How We Prove OrgX Works.