The Gemma Family from Scratch — Lineage, Inventions, and Where It Fits

Google's open-weight LLM family, walked generation by generation. Distillation, local attention, effective parameters — the invention behind each release — plus the licensing detail that turns into an incident if you miss it, and how to actually run one on your own machine with Ollama.

Gemma: Open Models Based on Gemini Research and Technology

Restaurants and loaves of bread — what "open weight" means

ChatGPT and Gemini are restaurants. The food — the answers — is good, but you never get into the kitchen and you cannot take the recipe home. When the place closes, whether that means a shutdown or a price hike, that is the end of it.

Gemma, which Google has been shipping since 2024, is closer to handing out loaves that have already been baked. The recipe — the training data and the training procedure — stays private, but the bread itself, the trained weights, comes home with you. Slice it and eat it (inference), knead it back into dough and rebake it (fine-tuning), or serve it in your own shop (ship it inside a product). Models distributed this way are called open-weight models.

Gemma is the younger sibling of Gemini, spun out of the same research and technology, and it has shipped at roughly one generation a year — the first in 2024, Gemma 4 in 2026. This article walks that lineage in order: what each generation actually invented, the licensing detail that turns into an incident if you miss it, and how to put one to work on your own machine with Ollama.

Groundwork: how to read "7B", and what a family is

The "2B" and "7B" in a model's name are parameter counts. B is billion, so 7B means a gigantic function with seven billion tunable numbers inside it. Whether it fits on your machine comes down to how many bytes you keep each one in. At 16 bits (two bytes), 7B is about 14GB; compressed down to 4 bits (quantized), about 3.5GB plus overhead. So memory ≈ parameter count × bytes per parameter — one multiplication is enough to estimate what you need.

A "family" is the set of siblings — different sizes, different specialities — built from one design philosophy and one training pipeline. Within a generation the tokenizer and the conventions are shared, which makes it cheap to prototype on the small one and swap the large one in later.

Generation 1: Gemma 1 (2024) — entry and foundations

The first Gemma arrived in February 2024 in two sizes, 2B and 7B. Inside was a standard decoder-only Transformer built from stacked attention, with an 8K-token context. Meta's Llama models already had a head start in open weights; this was Google entering with "small, high-quality models built on Gemini's technology."

The conventions set here are still in force. Every size ships in two flavours: a base version, the raw thing that does nothing but predict what comes next, and an instruct version — the one carrying the -it suffix — tuned to follow instructions. For chat you always want the -it side. This is also where the pattern of side branches growing off the trunk began: CodeGemma for code, PaliGemma for image-plus-language.

Generation 2: Gemma 2 (2024) — the invention of distillation

June 2024 brought Gemma 2 at 27B and 9B, with a 2B added later. Its defining feature is that the smaller models were trained by knowledge distillation.

The analogy: instead of studying alone from past exam papers and their single correct answers, you get to watch how your tutor hesitates. Ordinary training teaches one point — "the next word is cat." Distillation has the student copy the entire probability distribution a large teacher model emits: cat 70%, kitten 20%, dog 5%, and so on. The teacher's knowledge lives precisely in how it spreads probability across the candidates that are not the answer.

The knob controlling how smooth that distribution is is the temperature $T$.

In equation (1), $z_i$ is the raw score the model produces — the logit — and $T$ is the temperature. $T=1$ is plain softmax; raise $T$ and every score shrinks toward its neighbours until the gaps stop mattering, flattening the distribution out. In words: score every candidate word, divide each score by the temperature, push it through $\exp$ so nothing comes out negative, then divide by the total of all of them so the whole set adds up to 1 — percentages, in other words. Dividing by a larger $T$ before that comparison is what narrows the gap between first place and second, which says the same thing as the model gets less certain. Distillation turns $T$ up to smooth the teacher's distribution before the student copies it, pushing the information carried by the second-place and lower candidates through to the student.

What distillation buys translates directly into "smarter at the same size." Learning from a distribution a large model has already digested is more sample-efficient than a small model working through raw data on its own — and Gemma 2's 2B and 9B rode exactly that to a reputation for punching above their weight.

The second invention was making attention cheaper. Gemma 2 alternated, layer by layer, between global attention that surveys the whole sequence and local attention — a sliding window — that only looks at its immediate neighbourhood. The judgement call being made: surveying the entire text in every single layer is an extravagance.

Generation 3: Gemma 3 (2025) — it gets eyes, and it reads long

Gemma 3, from March 2025, comes in four sizes — 1B, 4B, 12B and 27B — and it is the generation that changed the most.

It grew eyes. From 4B up, a SigLIP image encoder is built in, making these multimodal models that accept images as input. The line that had been an experiment over in PaliGemma merged back into the trunk.

Context went from 8K to 128K. Every size but 1B can now read sixteen times as much at once. What made that possible was taking local attention seriously. The local-to-global ratio, 1:1 in Gemma 2, was pushed to 5:1 in favour of local, holding down the growth of the KV cache — the memory that keeps the intermediate results of past tokens. Global attention costs grow with the square of the context length, so long-context support is fundamentally a question of how you escape a quadratic curve.

The other change that mattered in practice was the official release of QAT (quantization-aware training) checkpoints. Dropping to 4 bits normally shaves quality off; shipping official builds that were further trained knowing they would be quantized meant the quality floor of running 4-bit locally became something Google supports on purpose. (The mechanics of quantization are covered in LLM Quantization from Scratch.)

Generation 3.5: Gemma 3n — enter the "effective parameter"

The same year, 2025, brought Gemma 3n, aimed at phones and edge devices. This is where the odd size labels E2B / E4B first appear. The E stands for effective.

The trick is best pictured as a matryoshka doll. A small doll (the effective parameters) nests inside a large one (the total parameters), and on a device short on memory you can run just the inner, smaller model — the MatFormer design. The total parameter count can exceed 5B while memory and compute behave like a 2B model; that is what the E2B label is telling you. Audio input also entered the family through this line.

Generation 4: Gemma 4 (2026) — the E-series takes over, and the license flips

Gemma 4, released at the end of March 2026, promoted the effective-parameter idea 3n had been experimenting with into the main line. The small end is E2B and E4B; the large end lines up MoE (mixture-of-experts) configurations that only ever use a fraction of their parameters. It is a generation that advertises a total parameter count and runs at an effective cost. The other big shift is the move to Apache 2.0, which the next section covers.

On my own machine — an M1 Max with 64GB of memory — Ollama's E4B (about 10GB at 4-bit quantization) is what I keep running day to day. Its Japanese reads naturally, it holds steady when told to emit JSON, and it is good enough to put into real service as the API fallback for internal tooling.

Licensing — read what "open" actually says

This is the section that turns into an incident if you skip it. The license on Gemma 1 through 3 is not a standard open-source license like Apache 2.0 or MIT. It is a bespoke one — the Gemma Terms of Use — and while commercial use is permitted, it comes with strings: (1) you must comply with the prohibited-use policy, and (2) when you redistribute the model or anything derived from it, you are obliged to impose the same use restrictions downstream. Ship Gemma 3 inside your own product and Google's terms reach right through to your users. That is exactly why people say "open weight" rather than "open source" in the OSI sense.

Gemma 4 changes this and ships under Apache 2.0 — free to redistribute and modify, patent grant included, the software industry's standard license. The barrier to bundling and redistributing drops sharply. But as long as the license differs by generation, the right posture is not "it's Gemma, so we're fine" but read the terms for the generation you are actually using. Note too that what you may do with the model's outputs, and who holds the rights to the data you fine-tune on, remain separate questions that the model's own license does not settle.

Hands on: running it locally with Ollama

The shortest path to local is Ollama. Two lines take you from nothing to a running model.

brew install ollama          # macOS; other platforms from the official site
ollama run gemma3:4b         # first run downloads automatically, then opens a chat

Tags pick the size and the variant: change the size with gemma3:27b, point at a QAT build with gemma3:4b-it-qat, jump generations with gemma4:e4b. Settings go in a Modelfile.

# Modelfile
FROM gemma3:4b
PARAMETER temperature 0.7    # output diversity (the T from equation (1))
PARAMETER num_ctx 32768      # context length; the default sits far below the model's ceiling

temperature is the $T$ of equation (1) repurposed for sampling at generation time — turn it down for safe output, up for varied output. The one that matters most is num_ctx. To save memory, Ollama defaults the context length to something far shorter than the model's ceiling (128K for Gemma 3). Paste in a long document and the overflow is discarded without a warning — this is the source of the single most common local-LLM complaint, "the model doesn't know the first half of what I fed it."

To use it over HTTP, OLLAMA_HOST=0.0.0.0 will accept outside connections — but Ollama has no authentication mechanism of its own. The rule is never to leave it bare on a public network; keep it behind a tunnel or a reverse proxy.

How this plays out in practice

Who reaches for it, and when. Three cases are typical. ① A backend or product engineer who needs to process data containing personal or confidential information without sending it to an outside API — with local inference, the data never leaves your hands. ② As the fallback for a commercial API's cost, rate limits or outages; that is exactly my own production setup, which on an API failure tries Ollama first and the commercial API second. ③ A mobile or embedded engineer wiring up on-device inference — the E-series in 3n and 4 exists for this.

What you actually touch. ollama pull/run; num_ctx and temperature in the Modelfile; OLLAMA_HOST and OLLAMA_ORIGINS when you expose the API; the choice between -it and base on Hugging Face; the QAT tags. The vocabulary in this article is the parameter names.

Pitfalls that turn into incidents.

• Using a base model for chat: it won't follow instructions, it will just start continuing your text. For conversation, always -it.
• The default num_ctx: as above, long inputs get truncated in silence. In a RAG setup that stuffs the context, this is the first place to look.
• Confusing generations with licenses: redistributing Gemma 1 through 3 carries the obligation to propagate the terms. Check the wording for your generation before you bundle it in a product or hand it outside the company.
• Quantizing too hard: go below 4 bits and it is usually multilingual fluency and structured output such as JSON that break before English does. If Japanese quality is a requirement, step the quantization back one level and compare.

If it comes up in an interview. "Why did Gemma 2 use distillation?" — because copying a large model's output distribution (how it assigns probability to the non-answers, smoothed by the temperature in equation (1)) is more sample-efficient than a small model teaching itself from raw data. Having read this far, you can answer it with the equation attached.

Wrapping up — the lineage is a record of fighting constraints

• Gemma 1: entry into open weights. Established the base/-it convention and the pattern of side branches
• Gemma 2: distillation delivers "smart for its size." The local-plus-global attention economy drive begins
• Gemma 3: image input and a 128K context. A 5:1 local ratio and official QAT builds make "it runs on your own machine" a serious commitment
• Gemma 3n → 4: effective parameters carry it onto devices. The license flips to Apache 2.0

Every one of those inventions is an answer to the same question: how smart can you get inside the limits of memory and compute? If you want to dig into what those limits really are — the quadratic cost of attention, and the KV cache — head to Attention from Scratch and The KV Cache from Scratch.