The Mistral Family from Scratch — Europe's Small-and-Strong Bet

How a Paris startup made 'small but strong' work: the sliding-window attention behind Mistral 7B, the sparse mixture of experts behind Mixtral, the Codestral code line, and the licence policy that started at Apache 2.0 and then wandered — explained from no prior knowledge.

A company named after a wind

The mistral is the cold, hard wind that comes down over southern France. In May 2023 a startup took that name and set up in Paris. Its founders were Arthur Mensch, formerly of Google DeepMind, and Guillaume Lample and Timothée Lacroix, who had built the first Llama at Meta.

The company's first model announcement was not a press conference or a blog post. In September 2023 it was a single BitTorrent magnet link posted to social media. That was Mistral 7B, released under Apache 2.0 — commercial use, modification and redistribution, all permitted with no forms to fill in. The licence was the statement.

This article follows the family along two tracks. The technical one: what actually makes these models small but strong. And the business one: how a policy that began as "everything under Apache 2.0" changed afterwards. The second track is where people get hurt when they ship, so it gets its own section later on.

Why "small" is worth anything

A model's parameter count translates almost directly into memory. Hold the weights at 16 bits and each parameter costs two bytes. A 70B model is therefore around 140GB of weights alone — you are lashing several datacentre GPUs together. A 7B model is about 14GB, and quantised down to roughly 4 bits it lands near 4GB, which is laptop territory.

So "13B-class quality out of a 7B model" is not a leaderboard flourish. It changes where you can run the thing and how many users you can serve at once. That is the target Mistral has aimed at consistently.

Inside Mistral 7B: two cuts to attention

Architecturally, Mistral 7B is the same decoder-only Transformer as the Llama family. The difference is two economies made around attention.

The first is GQA (grouped-query attention). While generating, a model has to keep the Keys and Values for every past token around (the KV cache). GQA lets several Query heads share one set of KV, cutting what has to be held to a fraction of the original. What this buys you is inference speed and memory, not accuracy.

The second is SWA (sliding-window attention). In ordinary self-attention every token looks at every other token, so compute and memory grow with the square of the sequence length $n$. SWA restricts each token's view to the previous $W$ tokens — 4096 in Mistral 7B.

Put in words, that says: instead of "each token looks at all $n$ of them", it is now "each token looks at $W$ of them". Because $W$ is a fixed number, the total cost stops scaling with the square of $n$ and starts scaling linearly with it — long inputs get more expensive in a straight line rather than a curve.

The obvious objection is that a 4096-token window should make anything further away invisible. But stacking layers widens the view. In layer one a token absorbs the previous 4096 tokens; in layer two it looks at tokens that have already done that absorbing, so information from about 8192 positions back is reachable. With $L$ layers:

which says that reach is roughly the window width multiplied by the number of layers. It is the same argument as the growing receptive field of a stacked CNN.

The wider landscape of attention variants, GQA and SWA included, is laid out in Attention variants.

Mixtral 8x7B: eight specialists, two on call

In December 2023 Mistral threw out another magnet link. This one was Mixtral 8x7B, also Apache 2.0.

The name invites you to multiply 8 by 7 and expect a 56B model. It isn't. The real total is 46.7B parameters, and only 12.9B of them are actually used to process any given token. That gap comes from a structure called MoE (Mixture of Experts).

Think of a large hospital. There are eight specialists on staff, but no patient is seen by all eight. The front desk — the router — reads the symptoms and calls in the two who fit. So the hospital pays eight salaries (memory), while any one consultation costs two doctors' time (compute).

Concretely, only the FFN (feed-forward layer) inside each Transformer block is split into eight copies. Attention is shared across all of them, which is exactly why the total isn't 56B. For each token, a small linear layer called the router scores the eight experts, the top two are selected, and their outputs are summed with weights.