Anomaly Detection from Scratch — Learning From Normal Alone

Why an AI can learn from good units alone on a factory floor where defect samples barely exist. Two rulers — reconstruction error and density estimation — built up from metaphor to equations, all the way to the part that decides everything in practice: choosing the threshold.

You can't collect the defects

An AI that sorts images into dogs and cats is trained on piles of dog photos and piles of cat photos. On a factory inspection line, where the job is separating good units from defective ones, can we use the same recipe?

Here manufacturing runs into a wall of its own. The better the factory, the fewer defects it produces. At a defect rate of 0.1%, collecting 1,000 defect images means producing a million units and having a human put a correct label on every last one. And defects are varied: scratches, chips, foreign particles, discoloration, misaligned printing — and worse, a way of breaking that nobody has ever seen may show up tomorrow. The premise that you can gather every defect pattern in advance and teach it simply doesn't hold.

So flip the idea around. Stop defining the defect, and study the normal exhaustively. Then treat everything that departs from the learned normal as suspect. That is the core of anomaly detection — specifically, the family of methods that train on normal data alone.

Metaphor: the appraiser who only ever handles the real thing

A currency appraiser doesn't come up memorizing a catalog of counterfeits. The heart of the apprenticeship, they say, is handling genuine notes, endlessly. The feel of the paper, the quirks of the printing, the ridge of the ink — once "normal" has soaked into the hands, even an unfamiliar type of forgery makes the fingers stop: something is off.

Anomaly detection from normal data alone is precisely an attempt to build that appraiser out of a machine. The point is not to enumerate anomalies but to trace the outline of normal with precision. Draw the outline, and anything falling outside it — however novel the anomaly — becomes detectable.

Intuition: two rulers for "how normal is this?"

There are broadly two ways to turn "departs from normal" into a number.

Ruler 1: reconstruction. Build a deliberately imperfect copier that can only copy normal units well. Feed it a good unit and a clean duplicate comes out; feed it a defective one and — since the copier never learned how to render anomalies — it fails to copy. The larger the gap between original and duplicate, the more suspicious the unit.

Ruler 2: density. Plot the features of normal images as a mass of points on a map, and chart where normal piles up. A new point landing in that crowded region is normal; one landing out in empty countryside is anomalous. The measure is whether you're near your peers.

Notice that neither ruler defines what an anomaly is at all. The only thing being defined is normal. Let's take them in turn.

Ruler 1: reconstruction error — a deliberately imperfect copier

The workhorse of the reconstruction approach is the autoencoder. An encoder squeezes an image down into a small piece of information (say, a few-hundred-dimensional vector), and a decoder restores the original image from it; both are trained on normal images only. Since we're working with images, the internals are built from convolutions (CNNs).

It amounts to "compress, then restore," so you can get a feel for the behavior from something familiar. JPEG is also compress-then-restore, and when you round harder, whatever couldn't be restored shows up as residue.

The anomaly score is that difference between original and reconstruction, nothing more.

Put in words: hand the picture to the copier, then measure how far the copy came back from the original. Nothing in this formula states what a defect is — it only asks how badly the copying went.

Here is what Equation (1) says. $x$ is the input image, $E$ is the encoder (compression), $D$ is the decoder (restoration), so $D(E(x))$ is the image after being compressed and then restored. The size of its difference from the original $x$ — square the difference at each pixel and add them up — is the anomaly score $s(x)$. Copied well means it looks normal; copied badly means it looks anomalous. That's the entire equation.

Why does it fail to copy the anomalies? The key is the bottleneck in the middle — how narrow the compression is. To crush an image of tens of thousands of pixels down to a few hundred dimensions, the decoder has no choice but to assemble images out of a vocabulary of parts that show up often in normal units: for this product, this screw at this position, a surface with this texture. A scratch or a foreign particle it never saw during training isn't in that vocabulary, so in the reconstruction that region gets replaced by what it would have looked like had everything been normal. Take the difference and the anomaly's location rises to the surface. Getting where the anomaly is on top of everything else is this approach's practical strength.