Paper walkthrough: BeaconKV — evicting KV cache by predicting when a reasoning model will re-read its own plan

Deep into a long chain of thought, a reasoning model occasionally looks back at the plan it wrote at the start. The queries that do this cluster into a handful of groups in embedding space — so keeping a few dozen cluster representatives (beacons) is enough to predict which KV pairs will be revisited. A training-free method with up to 5.8× less memory and over 4.3× more throughput, read through its equations and code.

BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference

Recent queries are a bad oracle for the future

The paper is "BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference" by Janghyeon Kim, Minsoo Kim, Kyuhong Shim and Jungwook Choi (Hanyang University / Sungkyunkwan University), arXiv:2609.04971, 4 September 2026.

The abstract in plain terms: Large Reasoning Models (LRMs) get their problem-solving power from long chain-of-thought (CoT) generation, but the key-value (KV) cache grows linearly with sequence length and for long traces often exceeds GPU memory. Existing KV cache compression estimates which tokens will matter later from recent queries, implicitly assuming they are a reliable proxy for future attention. The paper shows this breaks in long-horizon reasoning: some decoding steps produce Thought Revisiting Tokens (TRT) that re-attend to distant context, such as the task-solving plan written early in the trace. Systematic analysis finds that the queries behind TRT cluster into a small number of similarity groups in embedding space. BeaconKV, a training-free method, maintains beacon queries — compact representatives of each global query cluster — to anticipate which KV pairs will be revisited without storing the whole query history. Across four open-source LRMs and diverse reasoning benchmarks it generally outperforms existing compression methods, with up to 5.8× memory reduction while nearly preserving full-cache accuracy and over 4.3× higher throughput.

The metaphor: a long meeting, and the moment someone asks about the original plan

Picture a three-hour meeting. You are taking minutes, but your desk only holds so many sheets, so you periodically throw old notes away. The obvious rule is to keep whatever the last few minutes referred to, betting that the current topic will continue.

But every long meeting has a moment when someone asks, "wait, what did we decide at the start?" You need a note from three hours ago — and since it never came up recently, you threw it away long ago.

Reasoning models do exactly this, the paper argues: the look-back happens sporadically and unpredictably, and scoring by recent queries structurally misses it (§3.1).

Background: why the KV cache grows

Transformer decoding proceeds one token at a time, and each new token attends to the keys and values of every previous token. Those past K and V never change, so they are stored and reused — that is the KV cache (see KV cache from scratch for the basics). In the paper's §2.1 notation, the output at step $t{+}1$ is

$$o^{\mathrm{attn}}_{t+1}=\mathrm{Softmax}\!\left(q_{t+1}\bigl[K\|k_{t+1}\bigr]^{\top}/\sqrt{d}+M\right)\bigl[V\|v_{t+1}\bigr]$$

In words, compare the single new query q_{t+1} against all stored keys K plus the new key k_{t+1}, and mix all values with the resulting weights. M is the causal mask that hides the future, and d is the head dimension.

In words: compare the single new query $q_{t+1}$ against all stored keys $K$ plus the new key $k_{t+1}$, and mix all values with the resulting weights. $M$ is the causal mask that hides the future; $d$ is the head dimension. Every step appends $(k_{t+1}, v_{t+1})$, so the cache grows with generation length.

The paper's number: Qwen3-4B generating 32K tokens at batch 16 needs over 77 GB of KV cache alone, nearly filling an 80 GB GPU (§1). Average outputs on AIME24 and LiveCodeBench exceed ten thousand tokens (Appendix A.2), so this is not hypothetical.

Prior methods: score with recent queries, keep the top-k

Eviction-style compressors such as SnapKV, RPC and R-KV share one framework (§2.2). When the cache hits its limit, take the queries from the last $N_{\text{obs}}$ steps (the observation set $Q_{\text{obs}}$) and compute each one's attention weight over every cached key:

$$w_{\tau,j}=\frac{\exp\!\left(q_{\tau}^{\top}k_{j}/\sqrt{d}\right)}{\sum_{l}\exp\!\left(q_{\tau}^{\top}k_{l}/\sqrt{d}\right)}$$

In words, this is the fraction of attention that the query at step τ pays to key j, out of the total it spends across all keys.

Simply "the fraction of attention the query at step $\tau$ pays to key $j$." Take the max or mean over the window as importance $s_j$ and keep the top $B_{\text{KV}}$ entries (the wider landscape is in KV cache compression explained). The one thing to hold onto: the only judges are recent queries.

The observation: Thought Revisiting Tokens

In §3.1 the authors run R1-Distill-Qwen-7B on an AIME24 problem and visualise layer 18, head 16. Most queries between tokens 1066 and 1092 look at nearby keys (roughly 900–1092) — "local queries." But tokens 1068 and 1090 redirect attention to a distant region around tokens 100–450, where the text (Figure 2, Appendix E) contains the problem constraints and the solving plan drafted at the start. These are Thought Revisiting Tokens (TRT), and their queries are "global queries."

Quantitatively, each query's mean distance to its top-K attended positions (K=150) puts local queries near zero and global queries in a clearly separated distribution (Figure 1(b); Appendix B calls a query local if that mean is ≤ 200). Global queries are not confined to one layer or head — they appear sporadically across the model (Figure 3).

This is what breaks prior methods. TRT are sporadic and unpredictable, so a global query may land in the last 32 queries by chance, but which distant KV pairs will be re-read next cannot be estimated systematically from recent queries. KV pairs that matter later get evicted early, and accuracy falls faster as the budget tightens (§3.1).

The key insight: global queries form clusters

If global queries were scattered at random there would be nothing to do. §3.2 finds otherwise. Measuring cosine similarity between queries before RoPE (geometry without positional effects), tokens 1068 and 1090 are dissimilar to their local neighbours yet strongly similar to each other, and a PCA projection (Figure 4(b)) shows global queries falling into a small number of clusters rather than spreading out.