Paper Explained: FreeToken — Treating Your Own PC as a Single Elastic Inference Platform

FreeToken is an edge-native serving system for frontier-scale MoE models on personal hardware. Its centerpiece is a q* policy that decides how many missed experts to ship over PCIe versus execute in place on the CPU — using nothing but two measured bandwidths.

FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution

"The weights are open" is not the same as "you can run it"

A recipe published for free doesn't help if you don't own a commercial oven. Releasing parameters determines only who can obtain a model, not who can afford to run it — that distinction is where the paper starts (§1). And as agents drive inference demand upward, the cost lands hardest on individuals and small teams. The result: the capability gap is closing faster than the accessibility gap.

Yet more than a hundred million consumer machines already carry discrete GPUs. Steam alone reports over 200 million monthly active users, with discrete NVIDIA GPUs in roughly 72% of surveyed systems (§1). What's missing isn't hardware — it's a serving system that can treat a wildly heterogeneous consumer machine as one unified inference platform. That is FreeToken's premise (published 2026-08-17).

MoE opens one door and closes another

An MoE (Mixture of Experts) layer holds $E$ experts and routes each token through only $k \ll E$ of them. (For the mechanics, see Mixture of Experts.)

The paper's example makes it concrete. DeepSeek-V4-Flash has 256 routed experts in each of its 43 layers and activates 6 per token, so only 13B of its 284B parameters participate in any single token. At the deployed precision, that active footprint fits inside an RTX 5090's 32 GB (§1).

Here is the door that closes. Sparsity reduces per-token computation without proportionally reducing the memory needed to hold the complete expert pool (§1). The full set of expert weights still exceeds GPU memory by a wide margin, so inactive experts sit in host memory or on disk and enter the execution path on demand.

Wall 1: prefill destroys the sparsity

A decode step touches $k$ experts. Prefill pushes thousands of tokens through each layer at once, and the union of their routes activates nearly the entire expert set (§2.1). The working set that was supposed to be sparse becomes effectively dense.

In the paper's numbers, an FP4 deployment of DeepSeek-V4-Flash must transfer roughly 140 GB of expert weights, adding about two seconds on an RTX 5090 system (PCIe 5.0 ×16, ~60 GB/s), about five on RTX 4090/3090-class desktops (PCIe 4.0 ×16, ~25 GB/s), and ten or more on the ×8 links common in laptops — on every prefill. An engine that fetches experts on demand exposes that entire window as GPU idle time.

The second cost is re-prefill. Many recent models interleave full attention with sliding-window attention (DeepSeek-V4-Flash, GPT-OSS) or use recurrent layers (gated DeltaNet in Qwen3.6-35B-A3B, Kimi Delta Attention in Kimi-K3). Unlike a KV cache, these compress the whole prefix into one evolving state that cannot be partially reused, and since each saved state costs as much memory as hundreds of tokens of KV, engines keep only a handful of checkpoints. Meanwhile agent harnesses edit context on almost every turn — deleting old tool outputs, stripping thinking segments. Any checkpoint after the modified position becomes invalid, so the engine falls back and re-prefills thousands of tokens. An RTX 5090 delivers roughly a fifth of an H100's and a tenth of a B200's dense BF16 throughput, so each redundant re-prefill occupies the GPU for tens of seconds (§2.1).

Wall 2: nobody has a principled policy for serving misses

Decode is the opposite regime. Few experts fire per step, but if they aren't on the GPU, they must be moved or computed somewhere.

Existing engines freeze placement at load or prefill time: llama.cpp assigns MoE tensors to devices when the model loads, and KTransformers pins a "hot" subset in GPU memory and runs the rest on the CPU (§2.2). But routing shifts with every token, so a frozen placement captures only a small fraction of the routed traffic. The majority of expert evaluations fall to the CPU, leaving both the GPU and the PCIe link idle (§5.3).

Going CPU-only doesn't work either. At small decode batches, expert execution is memory-bound, and consumer platforms attach the CPU to just two DRAM channels — roughly 50 GB/s for dual-channel DDR4 and 80–90 GB/s for DDR5, against the 1–1.8 TB/s an RTX 4090 or 5090 draws from its own memory (§2.2). And the paper's key point: the right mixture is hardware-specific, and cannot be read off a spec sheet.

Wall 3: nothing on the edge is dedicated

The GPU is shared with the desktop compositor, browsers, and games, so the budget available to a serving engine differs across launches and can shrink or grow mid-session. The best split of that budget moves too: agentic sessions accumulate context, so KV demand grows while the expert working set stays roughly fixed, and a split chosen on the first turn is wrong many turns later (§2.3). Startup is expensive as well — reading a ~140 GB pool from a 7 GB/s NVMe drive alone takes about 20 seconds, before any warmup.

The design: a two-level expert memory

FreeToken's foundation is a simple hierarchy (§3). The CPU-resident expert pool holds the complete routed-expert weights and remains the source of truth. Non-expert weights stay resident on the GPU. Whatever GPU memory is left becomes a single elastic expert cache shared by all MoE layers. Each slot holds every tensor needed to evaluate one layer–expert pair, so residency, lookup, and execution all operate on logical $(\text{layer}, \text{expert})$ identifiers rather than tensor shards.

One property falls out of this and does a lot of work: because the source of truth lives on the host, GPU memory affects only performance, never correctness (§3.3). You can shrink or grow the cache at any moment and the answer doesn't change — which is what makes runtime reconfiguration possible at all.