ML System Design — The 90% Outside the Model

The accuracy you hit in a notebook is not a promise about production. Feature definitions, training-serving skew, monitoring that catches slow decay, and the retraining loop — the 90% that lives outside the model, laid out in the order you actually design it.

An analogy: reproducing a famous kitchen

A head chef perfects a sauce. Hand the recipe to ten branch locations and you get the same dish, right? You don't. Each branch buys from a different supplier, so the ingredients differ. If the prep instructions are vague, two cooks will do two different things. And if a walk-in cooler sits at the wrong temperature for half a day, nobody notices until customers complain. What determines whether the taste survives is not the chef's skill — it's how the kitchen is designed.

A production ML system has exactly this shape. A model that scored well in a notebook is a recipe, nothing more. Whether it keeps producing the same result tomorrow depends on whether the ingredients (data) arrive through a stable path, whether the prep work (feature computation) is identical at training time and at serving time, and whether anything is watching for the taste going off. Sculley et al.'s "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015) made this picture famous: draw the whole system as boxes, and the box labeled "ML code" is one small square in the middle.

This article is about everything around that square.

The shape: not a line, a loop

Textbooks draw the pipeline as a straight line — collect data, build features, train, evaluate, deploy. The moment you put it in production, the two ends join and it becomes a loop.

The deployed model makes predictions, those predictions change what users do, the changed behavior lands in the logs, and those logs become tomorrow's training data. Recommenders make this obvious: the items you showed are the only items that can be clicked, so they are the only ones that accumulate as positives. The model ends up learning from a world it created. Ordinary web services have no equivalent of this.

Around that loop, failures cluster at four boundaries:

1. Training vs. serving — does the same input produce the same features on both sides?
2. Past vs. present — did anything unknowable at prediction time leak into training?
3. Offline vs. online — does the validation number connect to a real outcome?
4. Model vs. world — will you notice when the world itself moves?

Designing an ML system is, essentially, putting a checkpoint at each of those four boundaries. Let's take them in order.

What an offline number actually guarantees

Start with boundary 3, because it sets expectations for everything else. A held-out set measures generalization inside one world: the stretch of history you sampled from. It says nothing beyond that.

Make a model more flexible and training error keeps falling while error on unseen data eventually turns around and climbs. You can push on that gap directly in the figure below. The point worth holding on to: even the "test error" drawn there is not production error. Test data was carved out of the past, so when the world moves, that number goes stale along with everything else.

So offline evaluation is a gate for "may this ship", not evidence of "this will work". For how to set that gate up properly, see overfitting and evaluation design. Whether it actually helps in production has to be measured online — which is the subject of statistical tests and A/B testing.

Features: you lose the moment the definition forks

The most common production failure isn't in the model — it's in feature computation. The classic version: training features are written in SQL on the analytics warehouse, and the serving path reimplements them in Python. Different author, different month, so one side treats a missing amount as 0 and the other imputes the mean. Nothing about the model is wrong, and yet production predictions quietly drift.