#linearity

1 articles

01 ·How Transformers Work·★ MEMBER·PAPER·9 min read Paper Deep Dive: Your Transformer Can Hold Two Thoughts at Once — Blend Two Texts and the LLM Still Remembers Both Average the embeddings of two unrelated texts, run a single forward pass, and an LLM's output still keeps both streams' next-token candidates near the top. We walk through the paper's evidence that this 'superposition linearity' fades during pretraining, comes back with light fine-tuning, and can be partially decoded into two continuations.