#looped-transformer

2 articles

01 ·Training & Alignment·★ MEMBER·PAPER·7 min read Paper Deep Dive: Transformers Stop Thinking Too Early — How a One-Layer Rank-8 LoRA Stretches 3-Line Reference Following to 50 Pretrained LLMs follow in-context variable references for only two or three lines. Add a rank-8 LoRA (about 65K parameters) at one early layer with every weight frozen, and Qwen3-8B goes from 15.5% to 99% on 24-line chains. We walk through the mid-layer 'relay' behind it and the sharp limit on where the fix can go. 02 ·Inference & Serving·★ MEMBER·PAPER·16 min read SMELT explained (arXiv:2609.01343) — compute-matched looping Does running the same layers twice help once the budget is matched? SMELT holds per-token FLOPs, total parameters and KV cache within about 4%, loops the middle half twice, and reports 6.8-18.0% training-FLOPs savings on the compute-optimal frontier.