Embeddings from Scratch — from word2vec Intuition to Contextual Embeddings

Why does 'king − man + woman ≈ queen' actually work? A from-zero tour of embeddings: the map metaphor, the distributional hypothesis, the word2vec math, numpy code, contextual embeddings after BERT, and the knobs you touch in real RAG systems.

Efficient Estimation of Word Representations in Vector Space

Turning words into coordinates on a map

Computers can't read. To a machine, the string "dog" is just a symbol — nothing in it says that dogs are anything like cats.

So we flip the problem: place every word as a point on a giant map. Words with similar meanings go near each other; unrelated words end up far apart. "Dog" sits next to "cat" and "pet", while "tax return" lives on the other side of the map. Once such a map exists, "similarity of meaning" becomes "distance on the map" — a quantity you can compute.

Those coordinates are what we call an embedding. A real map has two dimensions; meaning doesn't fit in two axes, so we use spaces with hundreds or thousands of dimensions. An embedding, then, is a word (or a sentence) converted into a few hundred numbers — a vector — in a way that preserves meaning. Every LLM, and every document lookup in retrieval-augmented generation (RAG), starts with this conversion.

The intuition: a word is defined by its neighbors

How do you build that map? Hand-labeling "dog is close to cat" one pair at a time is hopeless. The key is an old idea from linguistics, the distributional hypothesis:

You shall know a word by the company it keeps.

Fill in the blank in "I fed the ___" or "I took the ___ for a walk" — both "dog" and "cat" fit naturally. Words that appear in similar contexts have similar meanings. Flip that around, and it means you can build the map of meaning with no human labels at all: just count, over a huge pile of text, which words show up near which.

In 2013, Mikolov and colleagues at Google published word2vec, which implemented this hypothesis with a very simple neural network and showed you could learn high-quality word vectors from large corpora on an ordinary machine. That paper kicked off the embedding era.

The mechanism: word2vec just plays fill-in-the-blank

What the flagship training scheme, skip-gram, does is almost anticlimactic:

1. Pick a word from the text (the center word $c$, say "dog")
2. Try to guess the words around it (context words $o$, say "walk", "fed")
3. Nudge each word's vector a little so the guess gets better

Repeat a few billion times. To predict "walk" well, the vectors for "dog" and "cat" must produce similar answers, so words sharing contexts drift together on the map — the distributional hypothesis turned directly into a training loop.

Here is the "scoring rule" of the quiz as an equation:

In plain words: "the probability that word $o$ appears next to center word $c$ goes up as the dot product of their two vectors gets bigger." Here $\mathbf{v}_c$ is the center word's vector, $\mathbf{u}_o$ the context word's vector, and $\mathbf{u}_o^\top \mathbf{v}_c$ their dot product — a compatibility score. The denominator sums the score over every word in the vocabulary $V$; that's a softmax, which turns raw scores into probabilities that add up to 1.

One practical trick matters here. That denominator asks for a score over the entire vocabulary, which is brutally expensive when the vocabulary has hundreds of thousands of words. The follow-up paper replaced it with negative sampling: a binary quiz that only distinguishes the one true context word from a handful of randomly drawn fakes. Swapping "compare against everyone" for "spot the real one among a few impostors" cuts the cost by orders of magnitude.

Measuring "closeness": dot product and cosine similarity

Once the map exists, finding "words near dog" needs a ruler. The standard one is cosine similarity:

$$\text{sim}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\|\,\|\mathbf{b}\|}$$

Read aloud: "the dot product of the two vectors, divided by their lengths" — which is exactly the cosine of the angle between them. Same direction gives 1, perpendicular gives 0, opposite gives −1. It strips out length and compares only direction, i.e. the direction of meaning.

The party trick that made word2vec famous is vector arithmetic. In a trained space, compute $\vec{king} - \vec{man} + \vec{woman}$, look up the nearest neighbor, and out comes $\vec{queen}$ — a result reported in the original paper. Relations like "gender" or "capital-of" get learned as consistent directions of displacement in the space. Nobody ever told the model that the female counterpart of a king is a queen; the structure emerges from co-occurrence statistics alone. That's the charm of embeddings.

Semantic search in code (numpy)

The heart of nearest-neighbor search fits in a few lines:

import numpy as np

def nearest(query_vec, vectors, words, k=5):
    V = vectors / np.linalg.norm(vectors, axis=1, keepdims=True)
    q = query_vec / np.linalg.norm(query_vec)
    sims = V @ q                      # cosine similarity to every word at once
    top = np.argsort(-sims)[:k]      # top-k highest scores
    return [(words[i], float(sims[i])) for i in top]

Normalize every vector to length 1 up front, and cosine similarity collapses into a plain dot product — one matrix multiply scores the whole corpus. What a vector database does internally is, at its core, making this matrix multiply faster and cheaper on memory.

The one-vector-per-word ceiling, and contextual embeddings

word2vec has a structural flaw: each word gets exactly one vector. The "bank" in "bank account" and the "bank" in "river bank" collapse into the same point, so polysemy is lost.

Contextual embeddings broke that ceiling. In 2018, ELMo did it with an RNN/LSTM-based language model, and BERT with a Transformer built on attention: both read the whole sentence first, then compute each word's vector on the spot. The same "bank" now gets a different vector every time, depending on its neighbors. If word2vec is a printed dictionary, a contextual model is an interpreter who picks the right sense from context.

Today's mainstream goes one step further: sentence embeddings, which compress a whole sentence or paragraph into a single vector. BERT-family models fine-tuned so that "sentences with similar meaning land on nearby vectors" (Sentence-BERT and its descendants) are what power the retrieval half of RAG: embed the question and the document chunks into the same space, then match them by cosine similarity — precisely the toolkit this article just assembled.

How this is used on the job

The people who touch embeddings daily are ML and backend engineers building RAG, semantic search, or recommendations, plus data scientists evaluating retrieval quality. The decisions that actually come up:

• Model choice: an API model like OpenAI's text-embedding-3-small/large, or a self-hosted one like the multilingual-e5 family run through sentence-transformers. For non-English search, always check multilingual coverage first.
• Knobs you actually turn: the dimensions of the vectors (accuracy vs. index memory), the index's distance metric (cosine / dot / L2), and chunk size plus overlap when splitting documents. Vectors land in FAISS, pgvector, Qdrant, or similar.
• Pitfall #1 — mixing models: embeddings are only comparable when produced by the same model. Switch models and you must re-embed the entire corpus; embed only the queries with the new model and search silently degrades with no error anywhere.
• Pitfall #2 — normalization vs. metric mismatch: feed unnormalized vectors into an inner-product index and rankings can be driven by vector length rather than meaning. Check settings like normalize_embeddings=True together with the index metric, as a pair.
• Pitfall #3 — asymmetric model prefixes: e5-style models expect queries and documents to be embedded with different prefixes (query: / passage:). Skip the prefix and you won't get the performance the model card promises.

If an interviewer asks "when do you use cosine vs. dot product?", a solid answer is: "On normalized vectors they're identical. On unnormalized ones the dot product is affected by length, so for semantic similarity you use cosine — or normalize and use the dot product."

Takeaways

• An embedding is a coordinate for a word or sentence, arranged so that distance reflects meaning
• word2vec builds the map automatically by playing fill-in-the-blank, a direct implementation of the distributional hypothesis
• The rulers are the dot product and cosine similarity — identical once vectors are normalized
• Contextual embeddings (ELMo/BERT) broke the one-vector-per-word ceiling; sentence embeddings underpin RAG retrieval
• In production, the three classic accidents are mixed models, normalization mismatches, and missing prefixes

Next, head to RAG fundamentals to watch today's toolkit get deployed inside a real retrieval pipeline.