Versioning Data and Models — An Experiment You Cannot Reproduce Never Happened

If you cannot get "92% accuracy" back six months later, it was an anecdote, not an experiment. We build up content-addressing (naming things by their contents), lineage (the graph from inputs to outputs), and the three levels of "how much is worth pinning down" — from the hash math to a manifest you can write today to the mistakes that quietly ruin a year of runs.

Could you reproduce yourself from six months ago?

Anyone who has tried to write down a grandparent's recipe knows the problem. Soy sauce to taste. The usual pot. Cook it until the colour looks right. You follow the instructions exactly and it comes out wrong. The recipe is not badly written — most of what you need to reproduce the dish was never written down at all.

Machine learning notebooks are usually that recipe. "Trained with this script, 92% validation accuracy." Six months later the same command gives 89%. Maybe the dataset was refreshed. Maybe a library moved. Maybe the seed was never passed. Nobody can tell which, and so nobody trusts the 92% any more.

A result you cannot reproduce is useless for comparison and useless for improvement. "0.4 points better than A" only means something if A and B were measured under the same conditions. An experiment you cannot reproduce is an anecdote, not an experiment. This article is about the tools that turn anecdotes back into experiments.

Four inputs decide the output

Before deciding what to manage, it helps to be precise about what actually determines a training result. There are four things:

1. Code — model definition, training loop, preprocessing
2. Data — training set, validation set, and how they were split
3. Environment — PyTorch version, CUDA, cuDNN, OS libraries, even the GPU model
4. Randomness and order — weight init, shuffle order, dropout masks, the order examples arrive in

Exactly one of these is properly managed. Git was built for code; it does not see your hundreds of gigabytes of data, the contents of your container, or the random numbers drawn at runtime. So "I recorded the commit sha" records one quarter of the picture.

The other three leak in predictable ways. Data sits on a shared drive and someone quietly appends rows. The environment depends on "the machine where it worked", and that machine gets updated. The seed says seed=42, but changing the number of DataLoader workers changes the order examples arrive in. None of this is malice. It all comes from the same root: a name does not guarantee its contents.

Name things by their contents, not by their labels

The filename dataset_v2_final.jsonl promises nothing about what is inside. Someone adds a line and the name is unchanged; meanwhile the identical bytes may also live under train.jsonl and train_copy.jsonl. Because name and content are independent, recording the name does not let you reproduce anything.

So invert it. Derive the name from the contents. This is called content-addressing.

$$\mathrm{id} = H(\text{every byte of the file})$$

$H$ is a hash function such as SHA-256. It reads the file from start to finish and emits a fixed-length value — 256 bits, or 64 hex characters, for SHA-256. That $\mathrm{id}$ has three properties:

• Deterministic — same contents, same id, on any machine, at any time
• Sensitive — one byte different gives a completely unrelated value (similar contents do not give similar ids)
• One-way — you cannot get the contents back from the id

The id is a fingerprint. Say sha256:9f2b...c41a instead of dataset_v2_final.jsonl and the name finally guarantees the contents. The properties of hash functions themselves are covered in Hashing and Nearest-Neighbor Search.

"One byte different means a different id" is clear enough. What about the converse — can two different contents accidentally land on the same id? The birthday approximation gives us the number:

Here $p$ is the probability that a collision occurs somewhere, $n$ is how many distinct contents you have registered, and $b$ is the hash width in bits. When the exponent is tiny this reads as $p \approx n^2 / 2^{\,b+1}$. With SHA-256 ($b=256$), even a trillion objects sit over a denominator of $2^{257}$, so the probability is effectively zero. If the hashes match, you may treat the contents as identical — that is the working conclusion.

Whole directories get ids too. Hash each file, pair each hash with its path, sort the list so the order is fixed, and hash the list. That is a Merkle tree, and it is the same idea git uses for its objects. A dataset of a million files collapses into one 64-character line.

Fingerprints pay off when you compare. Finding duplicates inside a dataset by comparing every pair costs a number of comparisons proportional to the square of the item count; fingerprinting and dropping them in a table costs a number proportional to the count itself. That gap is not "a bit faster" — it changes the number of digits.