Paper explained: BDH-CQ — an AI that thinks without words. Recurrent memory plus latent reasoning resets ARC's cost frontier

BDH-CQ (arXiv:2608.09888) never verbalizes its intermediate reasoning: demonstrations are written into a recurrent memory, and the answer is computed by iterating in a high-dimensional latent space. A 150M-parameter model reaches 29.5% pass@2 on ARC-AGI-1 at $0.0007 per task, breaking the reported cost–accuracy Pareto frontier. We explain the paper using only its own text as evidence.

BDH-CQ: In-Context Learning with Recurrent Latent Reasoning

Thinking out loud is expensive

Today's reasoning models (think ChatGPT's reasoning mode) solve hard problems by writing out their intermediate steps as text — chain-of-thought (CoT). That design carries a structural tax: every intermediate state must be projected through a discrete vocabulary, emitted one token at a time autoregressively, and read back in before computation can continue. The longer the reasoning, the more tokens, latency, and compute it burns (§1).

The paper we're reading today, BDH-CQ (arXiv:2608.09888, from the research team at Pathway), takes the opposite road. Demonstrations presented at inference time are written into a recurrent memory on the spot, and the answer is worked out by iterative computation in a high-dimensional latent space. Nothing in between is ever put into words. With this design, a model of just 150M parameters reaches 29.5% pass@2 on the visual-reasoning benchmark ARC-AGI-1 at a computed cost of $0.0007 per task — less than a tenth of a cent — breaking through the previously reported cost–accuracy Pareto frontier (§1, §5).

ARC: a test of learning on the spot

ARC (the Abstraction and Reasoning Corpus) shows you a handful of input–output pairs of colored grids and demands that you infer the hidden transformation rule and apply it exactly to a new input. It was designed to measure skill-acquisition efficiency: not whether a system possesses a skill, but how little experience it needs to acquire one (§2).

Every answer is verifiable cell by cell, and every failure can be inspected visually. Because each task has several test inputs, you can distinguish "got one lucky hit" from "applies the rule consistently." As experimental material for studying in-context learning, it is about as controlled as it gets (§2).

An analogy: mental-arithmetic champions don't narrate

A soroban (abacus) mental-arithmetic champion moves an imagined board directly in their head. They beat someone who narrates every step — "add seven, carry the one…" — because they never pay the cost of converting intermediate results into words. The trade-off: nobody can see inside their head, so when they slip, you can only guess where from the final answer. That strength (speed, cost) and weakness (opaque process) is exactly BDH-CQ's character.

Mechanism 1: writing demonstrations into memory without touching the weights (§3.2)

A task provides $K$ demonstrations $D=\{(x_t, y_t)\}$ and a query input $x^\star$. BDH-CQ processes the demonstrations one by one, updating a recurrent memory $S$.

In plain terms: "mix the $t$-th demonstration $D_t$ into the current memory $S_{t-1}$ to produce the next memory $S_t$." The function $U_\theta$ decides how the mixing happens, and its weights $\theta$ stay frozen throughout inference. No gradient step ever runs; all of the "learning" happens inside this state variable. Think of $S$ as a notebook: the model rewrites one page per worked example, and the equation, in words, says that each new page depends on exactly two things — the previous page, and the example just read.

The paper positions this recurrent contextual state as playing the same role as the context-dependent associations that attention builds — with one decisive difference: there is no key–value cache that grows with context length (§3.2). Where a Transformer's KV cache balloons in proportion to the context, $S$ has a fixed size. Linear attention, the paper notes, is the conceptually simplest special case of this scheme: $S_t = S_{t-1} + U_\theta(D_t)$.

Mechanism 2: iterative reasoning in latent space (§3.3)

Once the demonstrations and the query have been ingested, BDH-CQ iterates inside a structured latent workspace $H$.

Reading it out: (1) encode the query $x^\star$ together with the memory $S_K$ into an initial state $H_0$; (2) apply the same transformation $F_\theta$ for $R$ rounds, refining the state; (3) decode only the final state $H_R$ into the answer $\hat{y}$. The division of labor — $S$ changes as evidence arrives and supports in-context learning, while $H$ is the workbench for the current query — is the skeleton of the system, and the intermediate states $H_r$ are never decoded into language. In everyday terms, $S_K$ holds the model's notes on the demonstrations and $H$ is the scratch pad for this one question; the middle equation is the load-bearing one, which says the scratch pad is rewritten $R$ times while the notes stay fixed, and only its final state is ever turned into an answer.

The foundation is BDH (the Dragon Hatchling), a post-Transformer architecture the same team introduced previously, built on high-dimensional positive activations, low-rank communication, and a recurrent associative state — "brain-inspired but not brain-imitative," in their own words (§3.1). One caveat the paper itself states plainly: the dimensions, exact update rules, and training recipe remain proprietary (§3.3, §4.1).

Training data and the headline result (§4, §5)

The 150M-parameter model is trained on an ARC-style mixture: the ARC-AGI-1 training set, RE-ARC, ConceptARC, ARC-Heavy, and ARC-GEN100K, combined with privately curated data and augmentations (§4.2). No evaluation-task demonstration pairs or task identifiers participate in training, and no parameters are updated at inference (§1).

On the 400-task public ARC-AGI-1 evaluation set, the system scores 29.5% pass@2 (24.25% pass@1), taking roughly 0.85 H200 GPU-seconds per task — $0.00070 per task at $3 per H200-hour (§5). Against ARC Prize's July 2026 data, the paper reports this as about 57x cheaper than GPT 5.6 Luna (Low) (34.2% at $0.040 per task), and still about 11x cheaper after accounting for that model's 80% price cut (§5). It also cites HRM/TRM — transductive systems that run optimization on the evaluation tasks themselves — at $1.48/$1.76 per task (§8). An independent black-box audit by co-authors from Bielik and NYU reproduced the 29.5% score without access to the model weights (§5).

What it can do, and where it stumbles (§6)

What makes this paper worth reading is that it doesn't stop at a single score — it dissects, with controlled experiments, what the model actually learned from the demonstrations.

A concept-by-concept profile. On ConceptARC (16 concept families x 10 tasks), the system reaches 59.4% pass@2 at the task level but 77.9% at the test-pair level. Families like ExtendToBoundary hit 9/10, while Copy and Order sit at 2/10 (§6.1).

The boundary of extrapolation. Using controlled tasks generated after freezing the model, the authors turn one difficulty dial at a time. Line propagation and motif copying stay perfect at 48/48 even beyond the demonstrated range, but ordering collapses from length six onward (1/24 at length eight), and nesting drops to 29/36 at depth five (§6.2). The decisive experiment is an intervention: adding a single demonstration at the test's complexity restores depth-five nesting from 19/24 to 24/24, and lifts length-eight ordering from 0/24 to 13/24 (§6.2, Table 3). The collapse was mostly a failure to extrapolate beyond what the demonstrations covered — not a lack of execution capacity.

Composition depends on the operation. A color mapping defined purely in context is applied perfectly — 96/96 held-out outputs even with eight simultaneous bindings. But composing operations splits sharply: rotation-plus-relocation succeeds 72/72, reflection-plus-relocation 47/72, and color-swap-plus-relocation 0/72 (§6.3, Table 4). "Reasoning difficulty" isn't one scalar, the paper argues — it's a bumpy, representation-dependent profile (§9).

Inconsistency. Of the 160 ConceptARC tasks, 52 had some but not all test inputs correct. If a rule had truly been induced, it should transfer to every test input of its task — so this partial success is evidence that the transformation is not yet applied consistently (§6.4).

Reasoning effort scales (§7, §9.2)

By exposing the model to several levels of latent-reasoning depth during training, the amount of "thinking" becomes a knob at inference time. Pass@2 rises monotonically — LOW 21%, MEDIUM 27%, HIGH 29.5% — with LOW cutting cost by 22% (§7, Table 5). The paper also reports early experiments confirming Transformer-like scaling laws in pretraining from 1B to 600B parameters (§9.2).

Limitations, and how to read the claims

• Not reproducible: internal dimensions, update rules, and the training recipe are proprietary. This is a measured system result, not an open, verifiable architecture release (§3.3, §4.1).
• ConceptARC is also in the training mixture (§4.2). The opaque-identifier replication rules out identifier/batch confounds, but the paper concedes it "does not rule out exposure through training or checkpoint selection" (§6.5).
• Cost comparisons are not apples-to-apples: the $0.00070 is the authors' computed hardware cost, while other systems' figures mix API prices and estimates (§5). Section 6.6 additionally quotes $0.00265 per task for the standard setting — a figure the text never reconciles with the headline number.
• pass@2 is not two independent attempts: the system returns up to two ranked candidates, and all 75 single-candidate records were already correct at rank one (§6.6).
• The generated-task analyses carry an authoring confound: the mechanic-stratified tasks were authored by GPT-5.6, and hand-written gravity tasks score 80–100% where the generated ones score 2.9% (Appendix A.2). The paper also flags the small samples — ten tasks per concept family (§6.1).

How this gets used in practice

What you take home from this paper isn't cheerleading for non-Transformer architectures — it's three lessons that apply to daily work.

If you write few-shot prompts. Table 3 translates directly into practice: include at least one example at the same complexity as your real inputs. Stack only easy examples and the model will be flawless inside their range and quietly collapse outside it (the gap between 0/24 and 13/24 was one demonstration). When evaluating, don't check whether one input passed — check whether the same rule carries across multiple inputs (the 78% pair vs. 59% task gap).

If you design inference budgets. The knob that corresponds to reasoning_effort or a thinking budget in commercial APIs behaves monotonically here too, 21% → 29.5% from LOW to HIGH. That's evidence for staged policies like "run everything at LOW, retry only the failures at HIGH."

If you read benchmark results. Three traps: (1) never plot self-measured hardware cost and API prices on the same axis; (2) always check what pass@k actually means — independent samples, or the top-k of a ranked candidate list; (3) ask for a failure profile by operation, not a single leaderboard point. This paper discloses all three about itself, which makes it a good model for how to read others.

Takeaways

• BDH-CQ writes demonstrations into a recurrent memory $S$ (in-context learning) and iterates in a latent workspace $H$ over a fixed-size state (latent reasoning) — never verbalizing intermediate steps
• At 150M parameters and $0.0007 per task it reaches 29.5% pass@2 on ARC-AGI-1, breaking the reported cost–accuracy Pareto frontier (reproduced by an independent audit)
• Controlled experiments pin down that demonstration coverage governs extrapolation, that composition is representation-dependent, and that consistent rule application is still unfinished
• The internals remain proprietary, and the paper itself lists the caveats: possible training exposure, non-comparable cost figures, and small samples

For the machinery underneath, see Attention from scratch, the lineage of stateful computation in RNNs and LSTMs, and the practical fight between context length and memory in the KV cache explained.