Memory is the structural lift — Phase 2 substrate benchmark

The April 17, 2026 Phase 2 experiment asked whether an agent can carry earlier decisions through a sequence of dependent tasks. Its archived matrix records 136 runs and an estimated model cost of $3.79 across four models and four configurations.

The reported pattern favored supplying prior state when later work depended on it. That is evidence about this task design, not proof that OrgX always outperforms a well-configured agent with access to the same information.

Make the dependency explicit

Each sequence had three tasks. In product work, a requirements document fed an engineering plan and then a launch thread. In incident work, a playbook fed a postmortem and runbook. In go-to-market work, a launch brief fed social content and outreach.

For example, the launch thread needed to reference a specific decision from the requirements document and avoid scope that document excluded. A model could write a plausible launch thread without seeing the document. It could not reliably recover that particular decision by guessing.

That makes the later task different from a self-contained writing test. The evaluator is checking both the output and its consistency with an earlier artifact.

Compare what each configuration receives

These configurations vary different inputs. Bare versus full changes memory, verification, and skills together; it cannot isolate the effect of skills.

The models were Claude Sonnet 4.6, Claude Haiku 4.5, GPT-4o, and GPT-4o mini. A separate Claude Haiku 4.5 judge assessed quality and consistency. Eight of the planned 144 runs were missing after a fetch failure.

The comparison deliberately changes access to prior state. A large difference between the bare and memory configurations therefore supports the importance of supplying that context; it does not isolate a unique advantage of OrgX over another system that supplies the same context.

What the result suggests

In the archived matrix, later dependent work generally scored worse without the earlier state. Critiquing a draft did not reliably compensate for missing source material. A verifier could identify that the launch thread omitted a decision, while the revision still lacked the document needed to recover it.

The full configuration did not improve every model or task. That comparison cannot identify skills as the cause: full adds both skills and verification relative to memory. A skills-only comparison would need the same context, verification, model, and task settings on both sides. This experiment does not provide that control.

The runner also began recording the actual skill bodies' versions and a content hash. That makes a later comparison more interpretable: a change to a skill should be distinguishable from a change to the model, prompt, or task.

The original article’s rounded cell-average table and chart do not reconcile with the checked-in 136-run matrix. This revision removes the chart and does not repeat those averages or present a new quantitative ranking. The original aggregation remains unresolved; the configuration table above explains the method without claiming a measured effect size.

The matrix and runner were private at the original publication. The public benchmark arena is not a substitute for a released bundle of this specific experiment. Readers could inspect the reported method, but could not independently reproduce these raw results from a public package.

Other limitations remain: one judge model, missing runs, style-sensitive scoring, and different context supplied across configurations. A rubric requiring verbatim preservation can penalize a useful paraphrase; that tests a stated requirement, but its relevance to the real task still deserves review.

Test the handoff on your own work

Choose a short sequence where the last output must honor an earlier decision. Keep the task and model settings fixed, vary how the earlier state is supplied, and inspect the resulting artifacts against that decision. Record failures and missing runs before comparing scores.

For OrgX, start with one recorded decision and ask a fresh supported session to retrieve its source before continuing. Cross-client recovery remains partial beta support. The methodology explains why a controlled result, an inspectable record, and a reliable production workflow need separate evidence.