The April 11, 2026 OrgX benchmark separated artifact generation from judging. It produced 45 artifacts and 135 judge calls, then failed the stricter quality gate on ten checks. That failure is part of the result.
The earlier smoke run asked the generating model to score its own work. That can test whether a runner works, but it gives a weak basis for claiming the work is good. Separate judge calls make the assessment more inspectable; they do not make it equivalent to human acceptance.
What ran
The published bundle records 15 tasks across seven domains, with three repeats per task. gpt-5-nano generated the artifacts. The recorded judge panel was gpt-5.4-nano, gpt-5.4-mini, and gpt-5.4, using low, medium, and high reasoning respectively.
Those are the model identifiers in this historical run, not a recommendation to use them today. The May follow-up describes the later panel repair.
| Check | Recorded result |
|---|---|
| Artifacts generated | 45 |
| Judge calls | 135, with no recorded failures |
| Normal bundle validation | Passed |
| Scorecard recomputation | Passed |
| Stricter quality validation | Failed on ten quality-bar misses |
| Human review recommended | 45 of 45 artifacts |
The bundle passed consistency checks while missing quality thresholds. Those results can coexist.
Open the result bundle to inspect the files behind those counts. A valid bundle and a recomputable scorecard establish that the record is internally checkable. They do not establish that the artifacts satisfy a publication-quality standard.
Follow one failure
The mobile modal interaction specification is a useful example. On its first repeat, the published assessment gave completeness 0.82, overall quality 80, and the engineering-ready criterion 0.75. Other design tasks also missed their thresholds.
The artifact existed and could be judged. The failure was that it did not clear the stricter bar. Reporting only task completion would have hidden that distinction.
To inspect the result, find the task in tasks.json, read the corresponding output in examples.json, then compare the criterion judgments. Decide whether the rubric is testing something a reviewer would actually need from that deliverable.
Design outputs in this run were Markdown handoff specifications, not rendered interfaces. A strong specification score therefore cannot establish that the eventual screen is usable, accessible, or visually correct. That would require a different artifact and review.
Why every artifact received a review flag
The flag was deliberately sensitive. It fired if judges materially disagreed, a judge recommended review, or a judge call failed. A flag was not itself a failed task; it meant the assessment should not be treated as settled.
Separate model calls are independent of the generating call, but their judgments may still share biases or mistakes. The panel needed calibration against human rulings before its scores could support stronger claims.
This is also why the failure list matters more than a single average. Averages can hide one domain's weak outputs. Inspecting a particular miss lets a reviewer challenge both the model's work and the benchmark's standard.
Cost and the remaining limits
The published accounting estimated 2.4623 cents for generation and 122.0173 cents for judging, totaling 124.4796 cents. Judging dominated the estimate. These are the bundle's model-usage cost calculations, not a general price for operating an autonomous initiative or a reconciled customer bill.
Future comparisons need the same accounting boundary, including reasoning tokens and failed judge calls. A cheaper generation model does not make the evaluation free.
The run demonstrated a reviewable mechanism: generate artifacts, collect separate criterion judgments, report disagreement, and retain failures. It did not demonstrate frontier organizational competence or reliable replacement of human review. Human baselines, rubric quality, design coverage, and judge calibration still needed work.
Read one artifact and its judgments before relying on the summary. The methodology explains the separate boundary between public mechanism runs and headline-eligible evidence. The next useful question is whether the recorded judgment holds up when a person inspects the actual work.
Sources & evidence Inspect the methodology and source files +
Research receipt
Inspect the run before trusting the claim.
The result files preserve outputs, judgments, assumptions, costs, and the scorecard behind this post.
- Version
- 2026-q1
- Tasks
- 15
- Domains
- 7
engineering · product · marketing · sales · design · ops · cross functional