Recommenders and Embeddings — Same Math as RAG, Different Goal

What sits behind "recommended for you" is very nearly the same math as RAG's vector search. A from-zero tour: matrix factorization, two-tower models, and how the ANN stack is reused — plus why the evaluation and the failure modes end up completely different.

BPR: Bayesian Personalized Ranking from Implicit Feedback

Who actually picks your recommendations

Picture the secondhand bookshop you keep going back to. The owner pulls something off the shelf and says, "You'll probably like this." They haven't read the book. What they've read is not the contents but what other customers with shelves like yours went home with.

That is exactly the idea behind collaborative filtering. You can infer taste from overlapping behaviour alone, without understanding any content at all. "Customers who bought this item also bought" is that sentence written out in plain English.

The opposite approach is content-based filtering, which actually reads the blurb and the genre to find similar things. RAG's document retrieval sits squarely on that side. Modern recommenders blend both inside the same equation, and what comes out is exactly the kind of meaning-bearing array of numbers covered in Embeddings from Scratch. If you've built a RAG system, you already know half of how a recommender works.

Intuition: filling in an enormous sparse table

Put users on the rows and items on the columns. The cells can hold ratings or just a clicked/not-clicked flag. A real service has millions of rows by millions of columns, and since any one person touches a tiny slice of the catalogue, almost every cell is empty. Predicting those empty cells is what recommendation means.

Why is prediction possible at all? If the table were random numbers, it wouldn't be. It works because a few dozen tendencies — "people who like sci-fi buy sci-fi" — already explain most of the cells. In mathematical terms, the table is low rank: it can be approximated by a tall matrix times a wide one.

Here $\hat{r}_{ui}$ is the predicted affinity of user $u$ for item $i$, $\mathbf{p}_u$ is a list of $d$ numbers describing that user's taste, and $\mathbf{q}_i$ is a list of $d$ numbers describing that item's character. The whole formula says: for each of $d$ hidden axes, multiply "how much this person wants it" by "how much this work has it," and add everything up. Said in words: $\hat{r}_{ui}$ is a single running total — walk down the person's taste list and the item's trait list side by side, and add a point wherever the two agree.

The crucial part is that nobody decides what the axes mean. No one defines a "sci-fi score." Run training so the fill-in error shrinks and the axes appear on their own — the same story as Singular value decomposition and low-rank approximation. And as a by-product, users and items become points in the same $d$-dimensional space. People and things land on one map. That is what an embedding is in a recommender.

Where plain matrix factorization loses to reality

This form was the star of the Netflix Prize era, but it will not carry a modern service as-is. There are three holes.

Cold start. A user who just signed up has an entirely empty row, so there is no material to learn $\mathbf{p}_u$ from. A title released yesterday has an empty column. And new titles are precisely the ones you most want to surface.

No door for features. The formula sees IDs and nothing else. Age, device, time of day, the item's own title text — none of them have anywhere to enter.

There are no five-star ratings. Very few people diligently leave stars; what actually accumulates is clicks, plays and purchases, i.e. implicit feedback. And it contains positives only. Whether a non-click meant "dislike" or "never appeared on screen" is something the logs simply do not distinguish.

How you handle that third one is where recommender systems break most often.

Implicit feedback, and how to manufacture negatives

You cannot learn like-versus-dislike from data that contains only positives. Negatives have to be manufactured, and how you do it drives the outcome. The standard trick is in-batch negatives: reuse the positive items of the other users in the same mini-batch as your negatives. Nothing extra needs loading, which makes it the default once the catalogue is large.

The other direction is to stop trying to predict absolute scores. BPR (Bayesian Personalized Ranking) learns only the pairwise ordering "what you saw ranks above what you didn't."

$$\mathcal{L} = -\sum_{(u,i,j)} \ln \sigma\!\left(\hat{r}_{ui} - \hat{r}_{uj}\right)$$

$i$ is the positive, $j$ the negative, and $\sigma$ the sigmoid. It says only that the loss shrinks when the positive scores higher than the negative — it never cares what the score actually is. Put in words, the formula asks for nothing more than this: whatever was clicked should sit at least one notch above whatever was not. Since the output is ultimately "the top 20," getting the order right is enough.

In-batch negatives do come with a side effect: popular items get picked as negatives more often, so they are pushed down again and again and sink unfairly. To cancel that skew, a correction that subtracts the log of the sampling probability from the score (logQ correction) is commonly applied alongside.

Two-tower models: giving factorization a door for features

The first two holes are closed at once by the two-tower model. The user tower $f(u)$ turns IDs, attributes and recent activity into a $d$-dimensional vector; the item tower $g(i)$ turns IDs, categories and the embedding of the title text into a vector of the same $d$ dimensions. The score is still a dot product.

$\tau$ is the temperature: the smaller it is, the more score differences get exaggerated and the sharper the training distribution becomes. The score, in words, is how closely the person-side vector and the item-side vector point the same way, with the temperature acting as a contrast dial on that reading. The skeleton is identical to $\mathbf{p}_u \cdot \mathbf{q}_i$. A two-tower model is just $\mathbf{p}_u$ and $\mathbf{q}_i$ with the ID-table lookup swapped out for a neural network's output. Arbitrary features now flow into a place that only accepted IDs, and a brand-new title still gets a $g(i)$ from its name and category.

So why split it into two towers at all? A single network taking both sides would clearly be more expressive. The answer is operational. Separate towers mean the item-side computation does not depend on the user — which means you can precompute $g(i)$ for every item and stuff it into an index. At request time you compute $f(u)$ once and search for neighbours. A model that consumes user and item together (a cross encoder) needs one forward pass per candidate, so it cannot generate candidates from millions of items.

That shape — a fast, coarse retriever narrows the field, a slow, smart model reorders it — is the same retriever→reranker split described in RAG fundamentals and design patterns. Different names, identical operational constraint.

The vector search stack is reusable as-is

Once training ends, what you hold is a pile of $d$-dimensional vectors. From here the plumbing is the same as RAG's: load every item vector into an ANN (approximate nearest neighbour) index and pull the top $k$ with the user vector. FAISS, ScaNN and HNSW are the same tools. Only the meaning of the similarity choice changes.

In RAG, vector length is usually noise from how a document happens to be written, so normalizing and using cosine is the natural move. In a recommender, though, length can carry the meaning of popularity: frequently purchased items get tugged repeatedly during training and grow longer. Normalize, and you throw that signal away with it.

Writing the two-tower loop in code

Training is startlingly short once you use in-batch negatives.

import numpy as np

def two_tower_scores(user_feats, item_feats, f, g, tau=0.05):
    U = f(user_feats)          # (B, d) user-tower output
    I = g(item_feats)          # (B, d) the positive items in this batch
    return U @ I.T / tau       # (B, B) diagonal = positives, off-diagonal = in-batch negatives

def in_batch_loss(scores, log_q):
    scores = scores - log_q             # strip the popularity head start (logQ correction)
    labels = np.arange(len(scores))     # row i's answer is column i
    return cross_entropy(scores, labels)

One line, U @ I.T, builds a $B \times B$ score table whose diagonal holds the only correct pairs. Each row is then just multi-class classification with "the answer is my own column," and you get $B-1$ negatives without loading a single extra example.

Serving is asymmetric. g runs as a batch job overnight; f runs online on every request — the two towers of one model live in completely different places.

index.add(g(all_items))              # precompute every item into the index
u = f(current_user)                  # once per request
cands = index.search(u, k=500)       # candidate generation: millions -> hundreds
cands = drop_unavailable(cands)      # drop out-of-stock, unlisted, already-seen
ranked = ranker.score(current_user, cands)[:20]   # fine-grained ranking

Same math, different goal

Recommenders and RAG share embeddings, dot products, ANN search and the two-stage split. What differs is the goal.

The right answer moves. "Which clause is that in?" has the same answer today as yesterday, but the film you want tonight is not last week's. So recommenders assume periodic retraining, and the index gets rebuilt on a schedule.

You learn from data you produced yourself. What you recommended becomes a log, and that log becomes the next training set. Fail to correct for position bias — clicked because it was placed on top — and whatever happened to be placed on top long ago stays strong forever. RAG retrieval has essentially no such loop.

Diversity is itself a metric. RAG may return nothing but highly relevant documents. A recommender that lines up twenty books by the same author has failed, not succeeded. Thinning out near-duplicates is a separate stage that sits after retrieval.

How this shows up on the job

Recommender and search engineers reach for this when standing up a "related items" or "recommended for you" rail on a new service. The usual order is to start content-based, with the item tower alone, while there are no logs, and add the collaborative side once behavioural data has accumulated. Trying to train a two-tower model from day one stalls for lack of material. MLOps engineers own the retraining and index-rebuild cycle; data engineers own the machinery (a feature store) that keeps training-time and serving-time features identical.

The knobs you actually turn: embedding dimension $d$ is easiest to start at 32–256, and raising it hits memory and latency directly. On the ANN side it's nlist and nprobe for FAISS IVF, or M, efConstruction and efSearch for HNSW — all of them trade recall against speed as you raise them. Candidate count $k$ reduces misses as it grows, at the cost of load on the downstream ranker. For libraries, implicit (ALS), LightFM and TensorFlow Recommenders are the usual entry points.

The pitfalls that turn into incidents, roughly in order of how often they bite:

1. Both towers live in one space. Refresh the user tower while leaving a stale item index in place and the dot product stops meaning anything. You cannot deploy one side alone.
2. Normalization mismatch. Trained on dot product, served on cosine. The definitions differ so the ranking differs, but if your offline evaluation runs through the same code path you won't notice — it only changes in production.
3. Survivorship bias in the logs. What never appeared on screen could never be clicked. Train on "not clicked = disliked" and anything without exposure stays buried forever.
4. Offline and online metrics diverge. recall@k improving on historical logs while CTR does not is entirely ordinary. Leave the final call to the A/B test.
5. Deleted things come back. Unlisted or out-of-stock items still sitting in the ANN index. Always filter after retrieval, and over-fetch to cover what you'll drop.

Design reviews usually press on two questions. "Why two towers?" — to precompute the item side into an ANN index, reserving the cross encoder for the final stage where accuracy matters. "Offline it won, but it lost the A/B — what do you suspect?" — position bias, a feature leaking future information, or a mismatched evaluation window.

Summary

• Collaborative filtering is really "a low-rank approximation of an enormous sparse table," and as a by-product users and items become points in one space
• A two-tower model adds a door for features and reshapes it so the item side can be precomputed
• What comes out is just embeddings, so RAG's ANN search tooling carries over unchanged
• The difference is the goal: the target moves over time, you train on your own logs, and diversity counts as part of the score