Memento 3 explained: a frozen LLM that learns by rewriting its own rulebook

The ARC-AGI-3 games were built to humble language models: at launch in March, the best frontier system scored half a percent on the hidden set. This week a team from University College London, Huawei's Noah's Ark Lab and the University of Liverpool posted an agent that clears every level of the 25 public games at the metric's ceiling using fewer than half the actions a human needs, and the language model inside it never changes. The interesting part is not the score, which others have matched. It is where the learning goes: into a plain-language rulebook and a Python program that the agent keeps rewriting, testing and shrinking.

The plain-English version

Imagine being handed a board game from another planet. No box, no manual, no one to ask. You can press buttons and watch the board change, and that is all. To win, you have to figure out what the pieces are, what the buttons do, and what "winning" even means, using as few presses as possible. That is what ARC-AGI-3 asks of an AI. Humans do fine. Ordinary chatbots did terribly when the benchmark launched, scoring under 1%.

Most AI systems learn in one of two ways. Either someone retrains them, which is slow and expensive and happens in a lab, or they keep notes in their chat history, which is cramped and gets thrown away. Memento 3 does neither. The AI model itself, Anthropic's Claude Opus 5, is frozen. What learns is a pair of files sitting next to it.

The first file is a rulebook, written in ordinary sentences: "there is a blue block that moves one square when I press right; the level ends when it touches the green square." The second file is a small computer program that the AI writes from the rulebook. The program is a simulator: give it the current board and a button press, and it predicts the next board.

Here is the key move. Before the agent presses a real button, it uses its simulator to search for the shortest route to the goal, like planning a route on a map. Then it acts. If the real board does something the simulator did not predict, the agent stops. It asks itself: was my rule wrong, or did I just write the code badly? It fixes whichever is at fault and rebuilds the simulator.

A new simulator is not trusted just because the AI feels good about it. It must pass two tests. The AI must agree the code actually says what the rulebook says. And the code must replay every button press the agent has ever made, from the start of the game, and reproduce every board it saw, square for square. A wrong rule can hide for a while, but it cannot survive that replay.

Think of a scientist with a lab notebook and a working model. Every experiment either confirms the model or forces a rewrite, and the notebook stays readable by anyone. That is the whole idea, and the result on the public games is that the agent used 7,518 button presses where humans needed 17,135.

How it actually works

The problem ARC-AGI-3 poses is rule discovery under a budget. Each game is a 64 by 64 grid, a handful of actions, and a sequence of levels. The agent sees frames and a signal when a level is complete, nothing else. The ARC Prize technical paper says the games rely only on "Core Knowledge" priors such as objectness, basic geometry and intuitive physics, with no text, numbers or cultural symbols. Scoring punishes waste: every action counts, including failed experiments.

The old way, which the paper's authors trace through their own earlier work, was to remember. The original Memento stored past cases in episodic memory and retrieved similar ones to improve a research agent. Reflexion-style agents store written reflections. Both help, but a memory of what happened does not tell you what will happen in a state you have never visited.

The new idea, which the authors call "Code as Model", splits knowledge into three artifacts:

1. The history. Every action and the frame that followed, kept in full.
2. The rulebook. A Markdown file holding hypotheses about objects, controls, dynamics, goals and what ends a level. Crucially, it may leave things unspecified. "The red tile does something when touched" is a legal rule.
3. The engine. A Python module compiled from the rulebook that predicts the full next grid and whether the level is complete.

The agent then runs a five-stage loop after each real action:

Why does it work? Three reasons stand out from the paper.

Rules before code. A mismatch has two possible causes: a wrong belief or a bug. By making the LLM decide which, and by writing a belief change into the Markdown before touching the Python, the system keeps its explanation and its implementation separate. Code is cheaper to fix than a wrong theory, and a wrong theory is easier to spot in prose.

Evidence gates acceptance. The authors write that acceptance "is not based on language-model confidence alone". Every candidate engine must reproduce every frame ever observed. Language models are fluent and often wrong; a deterministic replay check is neither.

Shortest explanation wins. Among engines that pass, the agent prefers the one with the smallest combined description length of rulebook and code, and the LLM is asked to simplify. Shorter theories generalise better to the levels not yet seen. This is Occam's razor as a selection rule.

Planning is the easy part once the engine exists. The agent runs a shortest-path search inside the simulator to the predicted goal and takes the first step. If no path exists, which is common early in a game, the LLM picks an intermediate target that would resolve an open question in the rulebook, such as touching an unexplained object or revealing an unseen area, and the agent plans toward that instead. Execution halts the moment reality and prediction diverge, and the loop restarts.

The deep dive

Formal setup

The paper models each game as a deterministic, goal-directed partially observable Markov decision process, $\mathcal{M} = \langle \mathcal{S}, \mathcal{A}, \mathcal{O}, T, \rho, G \rangle$, where the agent observes frames and goal-completion feedback but never the source or rules. The ideal agent would maintain a Bayes-adaptive belief over states and candidate models. In practice the learning state at time $t$ is

the action-observation history, the rulebook and the compiled executable. The LLM parameters $\phi$ appear in the update operators but are constant across the run. The executable $\theta$ exposes an initialiser, a transition function, an observation function and a predicted goal set, and state is reconstructed by replaying the history from the first frame rather than carried as hidden variables.

The objective is to maximise the probability of reaching a goal within an action budget $B$, with total actions as a tie-breaker. Model construction, simulator rollouts and replay checks are free; only actions sent to the environment count, matching ARC-AGI-3's scoring.

Admissibility and model selection

A proposed pair $(\psi, \theta)$ is admissible only if two conditions hold. First, replay consistency: for every recorded step, running the engine forward from the reconstructed state must reproduce the next observation, which for ARC-AGI-3 means every one of the 64 by 64 cells of every subsequent frame. Second, rulebook fidelity: $\theta \in [\![\psi]\!]$, the set of implementations consistent with the rulebook, as judged by the frozen LLM reading both. Because rulebooks can be underspecified, $[\![\psi]\!]$ typically contains many programs.

Among admissible pairs the agent minimises description length,

which the LLM approximates through a simplification step that is itself re-verified. The paper notes that a prior proportional to $2^{-\mathcal{L}}$ combined with the deterministic likelihood yields a posterior whose maximiser is the shortest admissible pair, so the procedure is a maximum a posteriori estimate under an MDL prior. Verification plays the role of the likelihood: it zeroes out contradicted hypotheses while leaving the prior odds among survivors untouched.

Planning

For the chosen engine the agent computes shortest-path costs $V_\theta(x)$ to the predicted goal set by a Bellman recursion and follows the greedy descent, so each step reduces $V_\theta$ by one and plans cannot cycle. This is certainty-equivalent planning on a point belief. The paper derives the full Bayesian success objective to show that information-seeking actions carry value, but the implemented controller substitutes the exploration heuristic described above rather than solving the belief-space problem.

The population extension

A single model has a self-confirmation problem: it plans only through states it predicts, so an error in states it never chooses to visit is never exposed. The population extension keeps $N$ admissible pairs over one shared history. Members are grouped into behavioural classes by identical plan action sequences, not by code differences, so syntactic variants of one theory do not get extra votes. Each planning episode samples one class uniformly and follows it until success, falsification or replanning. Three operations maintain the set: propose (generate and compile a candidate, keep only if admissible), repair (apply the reflective update to every falsified member) and plan/act. The authors are careful to call the resulting mixture "a computational approximation to the remaining version space, rather than a calibrated Bayesian posterior".

Experimental setup and results

The harness runs Claude Opus 5 inside Claude Code at extra-high reasoning effort and starts from the codebase of baseline1, Sergey Rodionov's executable-world-model agent, which maintains, simplifies and verifies a persistent Python model before planning. All numbers below are reported by the Memento 3 authors on the 25 public games (183 levels).

Per-game action ratios against humans range from 0.20 to 0.67 with a median of 0.42. Because the per-level score is $\min(1.15, h/a)^2$ under ARC Prize's scoring rules, where $h$ is the upper-median human action count and $a$ the agent's, anything beyond 15% better than human saturates. Action counts, not RHAE, separate the top two rows.

Ablation. Removing the rulebook and keeping only the code on three games (ft09, ka59, cn04, one per difficulty band) raised total actions from 617 to 677 and agent turns from 678 to 830. Both arms still scored 100. The rulebook is therefore worth about 9% in actions and 18% in LLM turns on this small sample.

Population. On wa30, the game with the largest human baseline (1,843 actions over nine levels), a two-model population cut actions from 899 to 597, improving eight of nine levels and losing slightly on level 8 (138 actions for the population versus 130 for the single model).

Atari Pong case study

To show the method is not ARC-specific, the authors learned a rulebook and engine for Pong in the Arcade Learning Environment with frame-skip 4. The engine fits latent quantities such as ball velocity and opponent phase from frames, and a feedback controller selects actions by simulating candidates. The LLM is used only during learning; evaluation runs with no LLM calls. Learning consumed 9,504 emulator frames. Three evaluation episodes with different openings all ended 21:0.

The authors concede that full-frame replay "retains residual rendering and paddle-motion errors"; the engine only needs to be accurate enough for control. Note the comparison is in frames, not actions, and the RL baselines' numbers are taken from their original papers, not re-run.

Closest prior work

The lineage is clear. WorldCoder (Tang, Key and Ellis, NeurIPS 2024) had an LLM write a Python world model and repair it against counterexamples, reporting orders-of-magnitude fewer environment interactions than deep RL on small games (a working Sokoban model after about 50 actions where deep RL needed over a million steps). WALL-E and AutoManual induced symbolic rules or manuals from prediction errors. Rodionov's baseline1 brought verified executable models to ARC-AGI-3, reaching 58.12% mean RHAE with GPT-5.5 in its May paper. Kepler, posted 30 September, reports a server-verified 100.00 with a frozen Claude Opus 5 configuration, 8,256 actions, 858 million tokens and $777.72 at list prices, with an unusual emphasis on documenting its own integrity failures.

What Memento 3 adds is the explicit two-level representation (prose rulebook plus compiled code, with the prose edited first), the conjunction of LLM fidelity judgment and exact replay as the acceptance rule, an MDL selection step, and the behavioural-class population. None of these is unprecedented alone. Together they are a tidy formalisation of a design the field has been converging on, and the ablation is the first attempt we have seen to put a number on the rulebook itself.

Why it matters

For everyday users, the direct effect is nil; nobody will play ARC games with this. The indirect effect is a preview of assistants that improve at your specific tasks between sessions without anyone retraining them, and whose "understanding" of your workflow is a document you could open and correct.

For developers, the paper is close to a design pattern. Most agent memory today is a vector store of past transcripts or a growing context window. Memento 3 argues for memory as a theory: a readable rulebook, an executable version of it, and a hard rule that nothing gets promoted to the executable without passing a mechanical test against everything seen so far. The ablation suggests the prose layer alone saves roughly a fifth of LLM turns, which is money. The population idea, keeping rival hypotheses alive and sampling among them, is a cheap defence against an agent confidently automating the wrong model of a system.

For companies, the auditability angle lands at a timely moment. Two days after the paper appeared, Microsoft chief executive Satya Nadella argued that "the controls that govern what a model can access and what actions it can take must sit outside the model". A world model that lives in a Markdown file and a Python module, with a replay log proving it is consistent with history, is a concrete example of learning that an auditor can read. Weights are not.

For the field, the headline claim of "recursive self-improvement" needs care. The authors define it narrowly: the agent autonomously revises a persistent world-model hypothesis, and the improvement is in prediction and planning, not in the LLM. That is a far cry from the sci-fi meaning. But it does reframe a live debate. ARC Prize itself has acknowledged that scores depend heavily on the harness, publishing GPT-6 Astra at both 62.7% and 99.9% on the same semi-private games depending on whether the harness let the model carry its reasoning state between requests. Memento 3 is evidence that the harness is not a nuisance variable; it is where a frozen model's learning lives.

What to be skeptical of

The ceiling was already crowded. Kepler reports a server-verified 100.00 on the same public set with the same base model; the Retrodict repository lists Tycho at 100.00, itself at 99.86 and baseline1 at 98.97; a community-leaderboard pull request for CCARC claims 100 with 7,809 actions. Memento 3's 7,518 actions is the lowest total we have seen reported for a full sweep, but it is not the first perfect score, and the paper does not compare against Kepler or Tycho's action counts directly.

Public is not hidden. ARC-AGI-3 has 25 public, 55 semi-private and 55 fully private environments. The public games have been studied by hundreds of harness builders since March, and Claude Opus 5's training data cannot be inspected for exposure. Kepler's README puts it bluntly: these are "public-set results, not held-out evidence". The ARC Prize is decided on the hidden set in a Kaggle sandbox with no internet and therefore no API model, which rules out this harness as submitted.

No cost, no variance. The paper reports no dollars, tokens or wall-clock time, saying only that each run "incurs substantial time and monetary cost". It does not state how many runs were made per game, so we cannot tell whether 7,518 is a typical result or a best one. Comparable harnesses cost between roughly $600 and $3,000 per sweep at list prices by their own accounting. The ablation covers three games and the population result one game.

The verifier is half LLM. Replay is exact, but rulebook fidelity is judged by the same model that wrote the code. A rule and its implementation could be jointly wrong in a way replay cannot detect until the relevant state is visited, which is precisely the self-confirmation problem the population only partially addresses.

Conflict of interest. Huawei's Noah's Ark Lab is a corporate research arm, and the backbone is a competitor's commercial model. We found no independent replication and no public reaction from ARC Prize staff as of writing. Coverage so far comes from newsletters that summarise the abstract.

Pong is a demo. The RL baselines are quoted from their original papers, the score is at the maximum where everything saturates, and the engine admittedly mispredicts some frames.

What to watch next

• 2 November 2026: final submission deadline for the ARC Prize 2026 ARC-AGI-3 track on Kaggle; winners announced 4 December. Watch whether any rulebook-style agent is adapted to run offline with open-weight models, since API models are barred.
• Code release. The arXiv page lists no repository. The earlier Memento papers were open-sourced; whether Memento 3's harness, rulebooks and replay logs appear will decide whether anyone can check the 7,518 figure.
• ARC Prize verification. Kepler's scores link to official server-replay scorecards. A matching scorecard for Memento 3, or a semi-private run, would move the result from self-reported to verified.
• Cross-harness action comparisons. Kepler, Tycho, CCARC and Memento 3 all sit at 100 RHAE; a controlled comparison of action counts and tokens with one fixed model is the obvious next paper.
• Rulebooks beyond games. The authors frame this as a general route to continual learning. Look for the approach on software or web environments where "cell-exact replay" has no obvious analogue.

Related reading from AI News Online: Gemini 4 Argon explained on why frontier labs are gating releases, and Anthropic's fake murder tip explained on what happens when agents that never give up meet the real world.

How this was made: researched and written by an AI model (Claude) from the primary sources listed above, then checked claim-by-claim against those sources in a separate AI fact-check pass. Spotted an error? Email [email protected] and we correct it publicly. Our process.