Paper Deep Dive: Transformers Stop Thinking Too Early — How a One-Layer Rank-8 LoRA Stretches 3-Line Reference Following to 50

Pretrained LLMs follow in-context variable references for only two or three lines. Add a rank-8 LoRA (about 65K parameters) at one early layer with every weight frozen, and Qwen3-8B goes from 15.5% to 99% on 24-line chains. We walk through the mid-layer 'relay' behind it and the sharp limit on where the fix can go.

Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It

A model that gives up three lines in

Write K = apple, B = K, D = B, ask for print(D), and the answer is apple. No outside knowledge is needed — you just follow the variables. Yet add a few more assignments and pretrained large language models start getting it wrong.

The paper is "Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It" (arXiv:2609.36585, Zehao Jin, Ruixuan Deng and Junran Wang, Georgia Institute of Technology, September 29, 2026).

Here is the abstract in brief. Pretrained transformers use little of their depth to follow references in context: thirteen base models reliably follow only 1.4–3.6 lines, and extra loops in pretrained looped models add little. A task-trained rank-8 LoRA at one early layer, with all model weights frozen, extends this computation. Qwen3-8B goes from 15.5% to 99% exact accuracy on 24-line chains, and a longer-trained LoRA reaches 50 lines. Ouro-1.4B reaches 60 lines after four loops and at least 160 after eight. What the LoRA starts is a relay: program lines pass on which chain they belong to through a short range of middle layers. Frozen heads read progressively further up the chain, and removing attention to the parent line stops the relay. A measurement on the frozen model locates the last useful intervention layer within tolerance in three of four held-out models, and task-specific LoRAs also improve MuSiQue. The authors' conclusion: default answers understate the computation a tiny edit can unlock.

Analogy: a relay that stalls at the third runner

Picture a queue. Only the first person holds "apple"; everyone behind knows just "mine is whatever so-and-so has." For the last person to answer, the fact "I'm on the apple line" has to be passed back. In the default model that hand-off dies after two or three people. The LoRA gives a small push at the start, but the passing itself is done by the people who were already standing there — the frozen layers.

The task and the fix (§2)

A program holds $c$ chains. The root assignment stores a single-token noun, and each later line names the previous variable. The right-hand variable is the line's "pointer"; the line that defined it is its "parent line." Reach is the longest chain followed with at least 80% accuracy.

The fix is one equation.

In words: take the state $h$ entering a layer, pass it through almost unchanged (scaled by $s$), and write a little extra through a narrow 8-dimensional path $BA$. $A$ is $8\times n$, $B$ is $n\times 8$ ($n$ is the hidden width), and $s$ is a scalar. Where ordinary LoRA (LoRA from Scratch) adds a delta to a weight matrix, this one acts on the layer's input itself, independently at each token. All communication between tokens is left to the frozen attention and MLP layers. Only $A$, $B$ and $s$ are trained — 65,537 parameters in Qwen3-8B, under 0.01% of the model.