2026-08-25·FREE·9 min read

The DeepSeek Family from Scratch — Breaking In with MoE and Distillation

DeepSeek redrew the open-LLM map with four tools: MoE, MLA, GRPO, and distillation. Walk the V2/V3/R1 lineage from zero prerequisites, learn how to read that famous training-cost number, and find out what the distilled models are actually for.

#deepseek #open-llm #model-families #moe #reasoning #distillation

ModalitytextTaskmodel-guideModel familydeepseek

The company that invites everyone to every meeting

At one company, every employee attends every project meeting. Legal topic on the agenda? Accounting is still sitting there. Every meeting burns headcount-times-hours. At another company, only the handful of people who matter get invited — but the ones who weren't invited are still on payroll. You pay for everybody, every month.

The first company is an ordinary neural network (a dense model). The second is a Mixture of Experts (MoE). That distinction is the front door to understanding DeepSeek. Their whole pitch — "enormous, yet cheap per query" — comes from changing who gets invited to the meeting.

There's one more thing about this company. They had their veterans work through a huge pile of cases, showed the transcripts to the juniors, and got the juniors doing passable versions of the same work on their own. That's distillation, and it's exactly what the models shipped alongside DeepSeek-R1 are.

The lineage: V2 → V3 → R1

Three generations, roughly.

DeepSeek-V2 (May 2024) — 236B total parameters, but only 21B actually fire per token. This is the generation that shipped both the DeepSeekMoE layout and MLA, a rebuild of the attention mechanism.
DeepSeek-V3 (December 2024) — scaled to 671B total / 37B active, trained on 14.8 trillion tokens. Here the star of the show is the training stack: FP8, Multi-Token Prediction, and auxiliary-loss-free load balancing.
DeepSeek-R1 (January 2025) — built on V3-Base, grown a long chain of thought using reinforcement learning alone. Released under MIT, with six smaller models distilled from it shipped the same day.

V3 and R1 have both seen revisions since, but the skeleton is fully explained by these three.

Design decision 1: MoE — separating "total" from "active"

In an MoE model, the feed-forward layer inside each Transformer block is replaced by a collection of smaller feed-forward networks called experts. For every token, a router scores which experts should handle it, and only the top-k actually run.

C_{\text{token}} \propto N_{\text{act}}, \qquad M_{\text{weights}} \propto N_{\text{total}}

(1)

In plain words: the compute to process one token scales with how many parameters fired that time, while the memory you must keep resident scales with the total parameter count. For V3 that means compute like a 37B model, but you still have to hold 671B worth of weights. Every advantage and every headache of MoE falls out of that asymmetry.

DeepSeekMoE adds two twists. First, slice the experts finely. Picking 1 of 8 large experts gives you eight possible routes; picking 8 of 64 small ones gives you an astronomically larger set of combinations, so specialization can get much more granular. Second, add a shared expert — one that every token passes through unconditionally, which soaks up the general-purpose knowledge everyone needs. The remaining experts are then free to actually specialize. V3's MoE layers used one shared expert plus 8 chosen out of 256 routed ones.

The trouble is that routers drift. Once a few experts become popular, tokens pile onto them while the rest sit idle and never learn. The traditional fix is an auxiliary loss that pushes usage toward uniform — but that distorts the objective you actually care about (predicting the next token). V3 instead used auxiliary-loss-free load balancing: a small per-expert bias is added to the router's score and nudged up or down depending on whether that expert is currently overloaded or starving. The bias affects only which experts get picked, never the gradient, so the loss function stays clean.

FIG 1The distribution of scores the router assigns to experts. Sharpen it and the same few experts get called every time (load collapses onto them); flatten it and nothing specializes because anyone will do. Load balancing is the job of managing that tug-of-war.

If MoE itself is new to you, Mixture of Experts from scratch covers the mechanism in detail.

Design decision 2: MLA — folding up the KV cache

While generating, an LLM keeps the Key and Value vectors of every past token around — the KV cache. On long inputs, that's what hurts. Ordinary multi-head attention stores num_heads × head_dim × 2 numbers per token per layer.

MLA (Multi-head Latent Attention) doesn't store K and V at all. It compresses them into a single low-dimensional vector and stores that, expanding back out with a matrix multiply at use time. It is low-rank approximation, applied to the cache.

\underbrace{l \cdot 2\,n_h d_h}_{\text{standard MHA}} \;\longrightarrow\; \underbrace{l \cdot (d_c + d_r)}_{\text{MLA}}

(2)

Here $l$ is the number of layers, $n_h$ the number of heads, $d_h$ the dimension of one head, $d_c$ the dimension of the compressed latent, and $d_r$ a small extra allowance for position information (more on that in a moment). Put differently: a cache that used to grow with the head count is replaced by one thin vector whose size is independent of it. The V2 paper reports a 93.3% reduction in KV cache versus the team's own dense 67B model.

FIG 2Drop the rank and the number of values you have to store falls off a cliff, yet the original matrix still reconstructs surprisingly well. That trade is precisely what MLA applies to the KV cache.

One thing gets awkward: position encoding. RoPE rotates keys according to position, which doesn't commute with the trick of keeping everything compressed and folding the projections together. So MLA compromises with decoupled RoPE — it carves out a small dedicated slice of dimensions that carry RoPE separately. That slice is the $d_r$ above. For how to size a KV cache in practice, see the KV cache from scratch.

V3's training stack, and how to read the cost debate

What stands out about V3 is less the model than the training infrastructure. Matrix multiplies run in FP8 (8-bit floating point), with the fragile parts — accumulation, normalization — kept at higher precision. DualPipe hides communication inside the idle gaps of pipeline parallelism. And Multi-Token Prediction has the model predict the next two tokens rather than one, squeezing more learning signal out of each position.

That's where the widely quoted number comes from. The technical report puts total training at 2.788M H800 GPU-hours, and — assuming a rental price of $2 per GPU-hour — about **$ 5.576M**. The report itself states plainly that this covers only the final training runs, and excludes prior research and ablation studies on architectures, algorithms, and data.

Three cautions when reading it. First, it's an assumed rental price, not an actual invoice and certainly not the cost of owning the cluster. Second, V3 is an MoE with 37B active parameters, so its FLOPs are orders of magnitude below what training a dense 671B would take — reading it as "a 671B model for $5.6M" is simply wrong. Third, failed runs, data pipelines, and salaries are all outside the number. What remains true is that they designed for an order-of-magnitude-cheaper run and actually took it to completion. The impressive part isn't the dollar figure; it's getting large-scale FP8 training to stay stable.

R1: hit it with pure RL and reasoning grew

R1 starts from an experiment called R1-Zero. Take V3-Base, skip supervised fine-tuning entirely, and run reinforcement learning only. The reward isn't a learned model of human preference — it's rule-based: is the math answer correct, do the code tests pass, is the required output format respected. Because no reward model is in the loop, there's far less surface for the model to game, so reward hacking is much harder.

The optimizer is GRPO. It throws away the value function that PPO normally needs (a whole second model), samples $G$ answers to the same problem, and turns each answer's standing within that group into the learning signal directly.

A_i = \frac{r_i - \operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)}

(3)

$r_i$ is the score of the $i$ -th answer and $A_i$ is how strongly to reinforce it. In other words, the only question is whether this attempt beat your own other attempts at the same problem. No absolute yardstick is needed, so you save all the memory and compute a value network would have cost.

The result: with nobody teaching it to, the model's answers got longer, and behaviors like pausing mid-solution to double-check its own work emerged on their own. But R1-Zero is hard to read and mixes languages mid-answer. So the shipped R1 was cleaned up in four stages — a cold start on a small set of well-formed long reasoning traces, then RL, then rejection sampling the good outputs into another round of SFT, then a final RL pass. For the general picture of shaping language models with reinforcement learning, see instruction tuning and RLHF.

The distilled models — don't be fooled by the names

The DeepSeek-R1-Distill-* models released alongside R1 are not R1. They are existing Qwen2.5 and Llama models supervised fine-tuned on roughly 800K answers collected by having R1 solve a large pile of problems. No RL was applied to them. Six sizes shipped: 1.5B, 7B, 8B, 14B, 32B, and 70B.

The paper's key finding is that distilling from a large reasoning model beats running large-scale RL directly on a small one. Reasoning patterns are much cheaper to imitate than to rediscover.

Two practical consequences. One is ease of deployment: architecturally these are Qwen and Llama, so vLLM, llama.cpp, and the rest of the existing stack run them unmodified. The other is licensing: R1 itself is MIT, but each distill inherits the terms of its base model — the Llama-derived 8B and 70B come with the Llama license attached. Check this before you make a redistribution or commercial call.

# A distill runs like any Qwen/Llama. Only the thinking tags need special handling.
from vllm import LLM, SamplingParams

llm = LLM(model="deepseek-ai/DeepSeek-R1-Distill-Qwen-14B")
out = llm.generate(
    ["<｜User｜>Solve 12x + 7 = 43<｜Assistant｜><think>\n"],
    SamplingParams(temperature=0.6, max_tokens=4096),  # leave room for the thinking
)
text = out[0].outputs[0].text
answer = text.split("</think>")[-1].strip()   # show only what comes after </think>

How this shows up on the job

If you run inference infrastructure and are sizing a deployment on your own GPUs, the total-versus-active asymmetry above is the first thing that bites. A V3-class MoE is 37B active, but the weights alone exceed 0.6 TB in FP8. This is not a single-80GB-card conversation. "It's 37B, so 4-bit quantization fits it on one card" is the classic accident that comes from confusing the two. The knobs you'll actually touch are --tensor-parallel-size and --enable-expert-parallel (vLLM), --max-model-len, and whether your backend has an MLA-aware attention path at all.

If you're wiring a reasoning model into a product, there are three traps. First, if you set max_tokens to a normal chat-sized 1024, generation gets cut off mid-thought and the user sees no answer at all. Second, <think>...</think> shouldn't just be hidden from users — the official repo recommends not carrying it into conversation history either; do so and the context fills with reasoning while behavior degrades. Third, the official guidance is a temperature around 0.6, no system prompt, with instructions placed in the user message. Greedy decoding (temperature 0) tends to fall into repetition loops.

If you're evaluating models, don't judge a distill by the number in its name. The 1.5B and 7B are built on Qwen2.5-Math bases, and they don't transfer gracefully to general tasks. And when you write "we ran DeepSeek-R1 locally" in an internal report, be explicit about whether that was the 671B original or a 7B distill. The world confused those two on a large scale in the weeks after release.

Wrapping up

MoE decouples memory (total parameters) from compute (active parameters). DeepSeek pushed on that with fine-grained experts, a shared expert, and auxiliary-loss-free load balancing
MLA folds the KV cache into a low-rank latent vector, which pays off directly in long-context serving cost
The famous training cost is misread unless you carry all three caveats: final runs only, an assumed rental price, and MoE-reduced FLOPs
R1 grew its reasoning from rule-based rewards plus GRPO. The Distill releases are not R1 — they're Qwen and Llama models SFT'd on R1's outputs, and their licenses follow the base model