schema-qwen

The benchmark: games that don't tell you the rules

ARC-AGI-3 (ARC Prize Foundation, March 2026) is a set of interactive games. The agent sees a 64×64 grid of 16 colors — that's the canvas on the left — and a handful of unlabeled actions: 1–5, sometimes a click 6@x,y, and RESET. Nothing else. No rule sheet, no object list, no stated goal, no reward signal. The agent must discover what the pixels mean, what its actions do, and what counts as progress, purely by acting and watching. Each game has several levels; the official metric (RHAE) scores completion and action-efficiency against first-time human baselines, squaring the penalty for wasted actions. Frontier models scored 0.51% at launch; the best official result by July 2026 was 13.33% (GPT-5.6 Sol, Public set). You can play the games yourself at three.arcprize.org — five minutes with one game makes everything below concrete.

The method we're reproducing: Schema

Schema (Impossible Research, July 2026) self-reports ~99% RHAE on the 25 public games using Claude Opus 4.8 + Fable 5 — with the same models scoring 42.8% under a generic coding harness. The claim is that the arrangement around the model, not the model, closes that gap. Schema makes the agent behave like a physicist:

• The world model is a program, not a vector. The agent's entire theory of the game lives in one editable Python file (right panel, live). It must define what the state is (init_state — which pixels form objects, what hidden variables exist) and how it moves (step(state, action)), plus render (state → expected grid) and is_goal (what completes a level). Because the theory is code, it is readable, diffable, and — the key property — executable: it doubles as a simulator.
• Certify against all of history. Every real transition ever observed is recorded in an append-only timeline. A backtest replays the candidate program over the entire record and demands exact, cell-perfect agreement. One wrong pixel = RED, with the counterexample. The agent cannot fool itself about how good its theory is.
• Plan inside the certified model. Once the backtest is GREEN, breadth-first search runs thousands of simulated games inside the program to find a shortest action sequence to the goal — costing zero real actions. This is where the efficiency comes from: pay for discovery once, then plan for free.
• Reality outranks the model. During execution every real frame is compared with the model's prediction; the first mismatch aborts the plan and becomes a counterexample the model must explain before planning resumes.
• Act to discover, not just to win. When several rules fit the history, the right move is the experiment that best separates them — commit it, observe, revise.

The loop you see in the play-by-play is exactly this cycle: observe → deliberate (theorize / backtest / plan) → execute → record, repeated per level.

Our reproduction

The harness (ursk/schema-qwen) reimplements the Schema loop from the blog post alone — no code was released. The games run locally via the official arc-agi toolkit; the agent talks to a local LLM served on this same Mac Studio. The harness owns everything deterministic: the append-only timeline, the sandboxed backtest with cell-level diff reports, BFS over the model's state space, per-step prediction checks during execution, and the persistent memory files (notes.md, world_model.py). The LLM owns exactly two things: writing the world-model code, and choosing which experiment to run next.

Where we deviate from the published method

1. A ~35B local model instead of frontier models. Schema used Claude Opus 4.8 / Fable 5 and GPT-5.6 Sol at max reasoning, with a two-model fallback pairing per game. We run one model: Qwen3.6-35B-A3B, 4-bit, on a single Mac Studio. Accordingly the goal is deliberately modest — fully clear one public game — not a 25-game RHAE score, and there is no fallback pairing.
2. Plain-text command protocol instead of native tool calls. Small models are fragile at structured tool-calling (our own SWE-bench-style evals showed illegal-tool-format as the dominant failure axis), so the agent answers with a python code block or a bare command line (PLAN, COMMIT …, NOTE:), parsed with regexes and re-prompted on failure.
3. The harness compensates for weak-model pathologies that the Schema post never needed to mention. Each was added after watching this agent fail in a specific way:
   • Repetition runaway — in long contexts the model can lock into repeating one line until the token cap; the streaming client detects periodicity mid-generation and truncates with an explanation.
   • Unchanged resubmits — the model narrates the right probe but pastes its old code back; identical resubmissions are rejected, and an explicit COMMIT outranks an unchanged code block.
   • Theorizing on thin data — with under ~12 recorded transitions, RED backtest reports nudge toward a real probe instead of another rewrite.
   • Regression blindness — every submission is scored (total mispredicted cells); the harness keeps the best-scoring model, announces regressions, offers REVERT, and starts each deliberation from the best-verified theory.
   • Richer counterexamples — mismatch reports include each cell's before-value and an aggregate breakdown ("N cells your model changed but reality did NOT / M cells reality changed but your model did NOT"), separating over-firing rules from missing mechanisms at a glance.
   Whether these are "deviations" or just the price of running the method on a small model is, in a sense, the experiment.
4. Strict certification, kept. Like the original, PLAN is only available on a fully green backtest — no tolerance for "cosmetic" mismatches. We briefly tried relaxing this (a validation run saw a frontier model locked out of planning all run by a 2-cell unmodeled counter font) and reverted the same day: the goal here is a fair-and-square clear, and every pixel is deterministic and therefore modelable. What we kept instead is a sharper counterexample: when all mismatches are confined to a few cells, the harness lists them and points out that whatever lives there can be decoded from recorded history.
5. Simplified planning. Our BFS searches simple actions plus clicks the model explicitly proposes via candidate_clicks(state), with node/depth caps, states keyed by canonical JSON. Schema describes richer deliberation-time search but not its exact machinery; ours is the minimal version.
6. Backtest semantics per level segment. We fold each level's transitions from that level's first observed frame; on level-up transitions we check is_goal instead of the (unseen next level) grid. The post doesn't specify its exact treatment; this is our reading.
7. No RHAE pipeline. We track raw action counts against the eventual goal of one cleared game. Any numbers here are self-measured and not comparable to official leaderboard scores.

How to read this dashboard

• Game — the real environment, live; uncheck follow live to scrub back through every recorded frame.
• Play-by-play — the run narrated from the structured event log: 🧠 model turns, ❌/✅ backtest verdicts with wrong-cell breakdowns, 🔍 BFS results, 🎮 executed actions (with SURPRISE / LEVEL UP flags).
• Agent notes — the model's own persistent scratchpad; its hypotheses in its own words.
• World model — the agent's current theory of the game as runnable code. Watching this file evolve — objects appearing, rules generalizing, representations being torn up after a counterexample — is the most interpretable view of the agent's understanding.

Status & caveats

Work in progress. The agent has not yet cleared a level; it is currently inducing the first game's mechanism (watch the backtest wrong-cell count trend down in the play-by-play). Everything is self-reported; the agent never sees the game's source code (the toolkit downloads it to run locally, but that directory is walled off from the agent). Expect the run to take many hours: one deliberation turn ≈ one local 35B generation.