AOLab / Field report / TOP-01
We gave three AI models the same incident-triage challenges, then changed how their agents coordinated. Some retained every decision. Others lost information, missed thresholds or reversed correct advice.
01 / Observed results
Hub, peer and adaptive teams each made six sequential model calls to triage twelve fictional incidents. Scores below are mean macro-F1 on matched seeds 102 and 103. Higher is better.
Small sample, clear boundaries. Only two scenarios have independently verified matched raw traces across all three models. Qwen was also run on seed 101; OpenAI and Claude seed-101 outcomes are historical aggregates and not included here.
OpenAI / dated model
Anthropic / dated model
Open-weight / OpenRouter route
These results describe particular model–workflow–scenario combinations, not a general ranking of models or proof that one organizational structure is superior.
Quality × cost / matched sample
Across six arms per model on seeds 102–103, Qwen cost less but had substantially lower mean decision quality in these runs. Provider pricing and billing evidence differ, so treat this as an exploratory comparison.
Linear cost bars, scaled to Claude. Qwen: provider response-reported cost. OpenAI and Claude: standard-rate estimates, not invoice reconciliations. These are not like-for-like billing measurements.
02 / Inside the failures
The traces show distinct failure mechanisms. They suggest where better organizational rules may matter—but do not yet establish causation.
In Qwen's peer workflow on seed 103, all three integration agents returned only one incident. The voting mechanism correctly processed that incomplete input, leaving eleven of twelve decisions unresolved. This was not an eleven-way voting disagreement.
GPT-4.1 mini's hub workflow misclassified an exact-threshold outage. Its auditor corrected the mistake; the final coordinator reverted to the wrong label. Peer integration classified it correctly.
Some Qwen specialist replies contained malformed JSON. Claude often wrapped valid JSON in Markdown fences that the existing parser could handle. Formatting errors may contribute to information loss, but have not been shown to cause it.
03 / The method
Three specialists saw different risk, operations and customer evidence. The organization then integrated their reports under a fixed triage rubric.
Research integrity
The verified primary comparison covers just two synthetic 12-incident scenarios, with one realization per model–workflow cell. Models and providers ran at different times; the Qwen inference provider was not pinned. Workflow prompts and deliberation also differ, so topology alone is not isolated.
TOP-01 does not establish population reliability, human organizational performance, or the superiority of a particular coordination graph. It gives us testable mechanisms for the next study.
| Model | Seed | Hub | Peer | Adaptive |
|---|---|---|---|---|
| GPT-4.1 mini | 102 | 1.0000 | 1.0000 | 1.0000 |
| GPT-4.1 mini | 103 | 0.9153 | 1.0000 | 0.9153 |
| Claude Haiku 4.5 | 102 | 1.0000 | 1.0000 | 1.0000 |
| Claude Haiku 4.5 | 103 | 1.0000 | 1.0000 | 1.0000 |
| Qwen3 235B | 102 | 1.0000 | 0.5387 | 0.9153 |
| Qwen3 235B | 103 | 0.6190 | 0.1333 | 0.6190 |
Quality: mean macro-F1. The 12 decisions within each scenario are correlated; they are not independent replications. The matched comparisons are descriptive, not statistical tests of superiority.
Download the sanitized result rows (CSV)The public CSV contains aggregate-derived per-arm scores only. Private trace archives and detailed prompts are not published. Exact methods and research collaboration: aolab@oxford.ai.
Challenge the experiment
Better questions, replication proposals and critical reviews are welcome.