◉ CLOSE READING
Language Models Can Control Their Own Attention
Language Models Can Control Their Own Attention
Abstract
Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply.
The whole paper is inside the “yet”. Note what is not claimed: nothing here says attention is inaccurate. The defect is that a correct mechanism pays a cost it does not need to pay, which is why the headline metric later is attended tokens rather than accuracy. “Must scan” is structural necessity, not obligation — with no way to know in advance which tokens matter, scanning everything is the only sound thing to do.
KV cache
The stored Keys and Values of every past token, kept so they are not recomputed each step. It removes compute and replaces it with memory traffic — which is the cost this paper attacks.
global attention
The layers that look at the whole context, as opposed to windowed or linear layers. Only these are billed per context token, and only these are what DA touches.
Background — Why “reading the KV cache” is the slow part
A Transformer consults the Keys and Values of every earlier token at each generated token; the KV cache stores them so they are not recomputed. The binding constraint is memory bandwidth, not arithmetic — a GPU multiplies matrices far faster than it can move bytes out of HBM.
At long context the moving dominates, and the cache read alone can consume as much bandwidth as loading the model weights. What this paper reduces is bytes read per decode step. Not parameters, not FLOPs.
A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn’t the model already know which parts of the context are relevant?
“Extrinsic” against “intrinsic” is the whole positioning, compressed into one pair of adjectives. The criticism of prior work is deliberately narrow — not wrong, merely still O(N) — which is the safest form of objection because it is a statement about asymptotics rather than about quality. The rhetorical question is unusual in an abstract; it is there to advertise that the idea is embarrassingly simple, which is the paper's real selling point.
proxy score
A cheap stand-in for the true attention score, used to guess which tokens will matter before paying for them.
extrinsic / intrinsic
Whether the selection comes from machinery bolted on outside the model, or is read off what the model itself produces.
To this end, we introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only).
“Elicits” is doing quiet work: not trains, not teaches. The claim is that the capability is already there and merely has to be asked for, which is exactly why every number later is framed as a floor. “Declarative” is borrowed from programming languages — state what you want, not how to compute it — and “protocol” rather than “method” signals that the contribution is an agreement between two parties, the model and the serving engine.
elicit
To draw out a capability that already exists, by prompting. Contrasted throughout with training it in.
chain-of-thought (CoT)
Reasoning emitted as text before the answer. Here the trace is not just explanation — it doubles as the control signal for the kernel.
The inference engine parses these declarations like tool calls and skips most of the KV cache read.
“Like tool calls” is not decoration, it is the reason the thing works zero-shot: tool-call syntax is the one structured format modern models have been drilled on in post-training, so no new notation has to be learned. “Skips most of” reads as hedging but is later cashed out at 52%.
Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale.
“Off-the-shelf” and “zero-shot” say the same thing twice on purpose: no weights were touched. The paired parentheses map onto the two models in order — a compression convention worth adopting. Note “pp”, not “%”: accuracy gaps are always in points, because a 2.75% relative drop and a 2.75pp absolute drop are different claims. The final relative clause is the paper's defence of its own weakest number.
zero-shot
No examples, no fine-tuning — the instruction alone. Here it is also the paper's excuse for every suboptimal behaviour it reports.
pp (percentage point)
The unit for a difference between two percentages. 87.01% to 85.74% is 1.27pp, not 1.27%.
DA unlocks a new axis of sparse attention, with further potential under training-based methods that future work can explore.
“A new axis” claims orthogonality, not superiority — and the paper honours that later by proposing to stack DA with the very methods it critiques. The closing formula (“future work can explore”) converts an admission that the result is incomplete into a claim that it is early, which is the standard last move of an abstract whose numbers are good but not decisive.
1 Introduction
Transformers compute attention over every preceding token at each decoding step, rendering them computationally expensive for long-context tasks… Specifically, Key-Value (KV) cache memory access latency heavily dominates decoding time in long-context regimes.
The participial “rendering them …” is how papers attach a consequence without spending a “so”. “Specifically” then narrows the claim by one notch — from computational expense in general to memory-access latency in particular — and that narrowing is the paper's whole territory. “Dominates” is not loose here; it is quantified in Section 5.4 as 73% and 86% of decode time.
For instance, in Qwen-3.5-397B-A17B…, roughly 15 GB of KV cache must be loaded per sequence at every decoding step for a 1M-token context; a memory bandwidth requirement comparable to loading the model’s 17B active parameters…
Note the choice of yardstick. 15 GB means nothing on its own, so it is converted into the one quantity every ML reader has an intuition for — the cost of streaming the model's own active parameters. The semicolon then appends a bare noun phrase rather than a new sentence, which is how you supply an interpretation without slowing the paragraph down.
active parameters
In a mixture-of-experts model, the weights actually touched for one token — far fewer than the total. The comparison only lands if you read it as active, not total.
Background — Why sparse attention is hard in the first place
“Just look at the tokens that matter” is the obvious idea. The obstacle is that you cannot know which they are in advance: attention scores exist only after the full attention matrix has been computed, so dropping low-scoring tokens requires first doing the work you wanted to skip.
Prior work therefore went in the direction of approximation — static rules first (keep the recent tokens, keep the historically salient ones), then per-step lightweight scans that pick a top-k. This paper sidesteps the circularity instead of approximating around it, by asking the model.
While sparsifying context attention emerges as a natural solution, a fundamental challenge remains: true attention scores are unknown a priori. Because these scores only become available after computing the full attention matrix, dynamically identifying which tokens to attend to is prohibitively expensive.
“A priori” is stronger than “in advance”: it says the ignorance is structural, not a matter of engineering effort. “Prohibitively expensive” is the field's fixed phrase for “technically possible, practically dead”. The While-A-but-B shape is the standard way an introduction closes off the obvious solution before proposing its own.
a priori
Before any computation or observation. The paper's entire problem statement lives in this phrase.
Conversely, more recent methods acknowledge the query-dependent nature of attention, approximating the mask via a lightweight scan over the KV cache at each decoding step… While this reduces the constant factor of the cost, the complexity per step remains O(N).
“Acknowledge” concedes that prior work already saw the problem, which keeps the objection technical rather than personal. The entire criticism is then one clause about asymptotics. Read it carefully before the contribution paragraph, because DA does not claim to remove O(N) reads — it claims to remove them as a per-step overhead, which is a narrower and more defensible thing.
By deriving the attention mask directly from the model’s generated reasoning trace, rather than approximating it via hidden activations, our approach eliminates the selection cost entirely. O(N) context reads remain only in the model’s declared global phases, not as a per-step overhead.
“By …ing, our approach …” is the standard one-sentence statement of a contribution, and the axis of contrast is spelled out explicitly: text the model wrote versus activations inside it. The second sentence is the part to imitate — immediately after the claim, the authors narrow it themselves. O(N) does not disappear; it stops being a per-step tax. A reviewer would have forced that sentence, and writing it yourself is cheaper.
reasoning trace
The text the model writes before its answer. Here it is not read by a human but parsed by the serving engine.
hidden activations
The internal vectors. Existing sparse-attention methods infer the mask from these, which is exactly the alternative being rejected.
§
Members-only from here
The rest of the close reading and the glossary are for members. $4.99/mo, cancel anytime.
Comments
Sign in to comment