The Llama Family from Scratch — The Main Line of Open LLMs

How Meta's Llama became the main line of open LLMs. The contrarian bet behind Llama 1, the lineage running through Llama 3.x, the llama.cpp / GGUF / Ollama ecosystem that grew around it, and the licensing traps to read before you ship commercially — starting from no prior knowledge.

The restaurant that published its recipes

A closed LLM like ChatGPT is a fine restaurant. You get to eat the food — the answers — but the recipe, the model's insides, never leaves the kitchen. You cannot borrow the kitchen, and you cannot adjust the seasoning to your own taste.

An open LLM hands out the recipe instead — or more precisely, the weights: the billions of parameter values a neural network ends up holding once training is done. Get the weights and everything else opens up. You can run the model on your own laptop or server, keep training it on in-house data, or take it apart to study how it works.

Since 2023, the centre of that open world has been Meta's Llama family. Get this lineage straight once and news about the other open models — Mistral, Qwen, Gemma — starts reading easily too, because most of them take Llama's design as their starting point.

Llama 1 (February 2023) — the contrarian bet: build it small, train it long

The received wisdom at the time was that capability follows parameter count. GPT-3 had 175 billion parameters, and every lab was racing to build something bigger. Then DeepMind's Chinchilla work (2022) showed that for a fixed compute budget you come out ahead by shrinking the model and feeding it more training data.

Llama 1 took that one step further. The stance: accept some inefficiency in training cost in exchange for a smaller model that is cheap to run at inference time, and train it on as many tokens as you possibly can. Four sizes — 7B, 13B, 33B and 65B — trained on up to 1.4 trillion tokens. The paper reported that the 13B model beat GPT-3, more than ten times its size, on many benchmarks. That was the moment "bigger is not everything" stopped being an argument and became a result.

The license was research-only and gated behind an application, but the weight files leaked onto the internet not long after release. Ironically, it was that leak that moved history. Within weeks came derivatives — Stanford's Alpaca (Llama 7B trained further on instruction-response data), then Vicuna, then more. What formed there is the prototype of today's open-LLM culture: as long as somebody builds the foundation, seasoning it — fine-tuning — takes very little compute.

Before the lineage: an LLM is a machine that outputs the probability of the next word

There is one operating principle shared by every Llama that is worth pinning down first. An LLM is a machine that outputs a probability distribution over what the next token — a fragment of a word — will be as the text continues. And the dial that controls how sharply peaked that distribution is is the temperature $T$.

Read it in words: take the raw score $z_i$ the model assigned to each candidate token $i$ — called the logit — divide it by the temperature $T$, then convert it into a probability $p_i$ with a softmax, the normalisation that makes everything sum to 1. Turn $T$ down and probability piles onto the top candidate, so the output goes rigid and stable; turn it up and the gaps between candidates flatten out, so the output goes varied and random.

The --temp option in the local-inference tools coming up is handing you that $T$ directly.

A look inside: a conservatively improved Transformer

Llama's architecture is a decoder-only Transformer — attention stacked many layers deep. There is almost nothing newly invented in it; what characterises it is that it soberly loads in every improvement already shown to work by that point. Three of them stand out.

The first is RMSNorm (pre-normalisation). Before a vector enters a layer, divide it by its own "typical magnitude" to put everything on a common scale.