2026-08-05·FREE·8 min read

Image Classification from Scratch — The Invention of the Convolution

Feed a photograph straight into a fully connected layer and you get over a hundred million parameters — and a model for which the same cat, moved a few pixels, is an entirely different input. The convolution solves both with one idea: slide a small window. Kernels, stride, padding and pooling, ending with softmax turning scores into probabilities.

#cnn #convolution #image-classification #pooling #softmax

ModalityimageTaskclassification

Feeding a photograph straight into a dense layer

The network we built in the previous article was a machine that multiplies an input vector by a weight matrix and pushes the result through a nonlinearity. So a photograph should work too: flatten a 224×224 colour image and you have a vector of $224 \times 224 \times 3 = 150{,}528$ numbers.

That straightforward idea breaks in two ways, and neither can be engineered around.

Break 1: the parameters explode

Put just 1000 units in the first hidden layer. The first weight matrix is 1000×150,528 — 150 million parameters, in the first layer alone. In 32-bit floats that is about 600 MB of weights, and during training a gradient of the same size plus the optimizer's internal state pile on top. With that many free numbers, the model also has ample capacity to simply memorise the training images.

Break 2: shift by a few pixels and it is a different input

The second break is the deeper one. Train on photos with a cat in the top left, then show the same cat in the bottom right. To a dense layer these two images are vectors with nothing in common: the weight responsible for a top-left pixel and the weight responsible for a bottom-right pixel are independent parameters with no relationship to one another. "A cat is a cat wherever it sits in the frame" is obvious to us, and the only way a dense layer can learn it is from data.

The two problems the convolution solved

The convolution solves both with a single idea. Take one small window and slide it across the image. That is all.

Say the window is 3×3, holding nine weights. Place it at the top left, multiply the nine overlapping pixels by the nine weights and add — exactly the $w^\top x$ from the previous article. Shift one pixel right and do it again. Run to the edge and you have a new grid of numbers.

Two constraints are baked into that operation.

Local connectivity — each output looks at a 3×3 neighbourhood rather than the whole input. In images, adjacent pixels are strongly related and distant pixels are barely related at all; that fact is written into the structure itself. Whether the input is 224×224 or 4000×3000, there are still nine weights, and the parameter explosion is gone.

Weight sharing — the same window is reused at every position. The nine weights that look at the top left are the same nine that look at the bottom right, so a vertical-edge detector learned in one corner detects vertical edges everywhere. That property is called translation equivariance.

This is not merely a trick for saving compute. The invention was encoding a prior — "this is what images are like" — directly into the architecture.

(f * k)(i,j) = \sum_{u=0}^{K-1}\sum_{v=0}^{K-1} f(i+u,\ j+v)\, k(u,v)

(1)

$f$ is the input image, $k$ is the $K \times K$ kernel (the contents of the window, also called a filter) and $(i,j)$ is the output position. What this says, in words, is that the dot product of the window's pixels with the weights is recomputed at every position. The two sums are just "run over the nine cells of the window"; $f(i+u,\ j+v)$ is the pixel the window happens to be sitting on, and $k(u,v)$ is the weight in that slot. Strictly speaking this is cross-correlation; the mathematician's convolution flips the kernel first. Since the weights are learned, the difference is only in the name.

Kernel, stride, padding

Three numbers get chosen in an implementation. Kernel size $K$ is the side of the window, and 3 dominates: two stacked 3×3 layers see the same span as a 5×5 while costing 9+9=18 parameters instead of 25. Stride $S$ is how far the window jumps each step; a stride of 2 roughly halves each output dimension. Padding $P$ is how many rings of zeros surround the input; with $K=3$ and $P=1$ the output keeps the input's size.

The output side length follows from these.

H_{\text{out}} = \left\lfloor \frac{H + 2P - K}{S} \right\rfloor + 1

(2)

$H$ is the input side and the outer brackets mean round down. Put in words, it counts how many times a $K$ -wide window fits along the padded input when it moves $S$ pixels at a time. The $2P$ is the width padding added, subtracting $K$ is what keeps the window from hanging off the far end, dividing by $S$ turns the remaining distance into a number of steps, and the $+1$ is the very first placement, before any step is taken. Most "dimensions do not match" errors are a miscalculation of exactly this.

Channels matter too. A colour image has three input channels (RGB), so the kernel has depth three, and 27 weights plus one bias produce a single output plane. That plane is a feature map — a map of where, and how strongly, the one pattern this layer looks for occurs. One pattern is never enough, so 64 or 256 independent kernels run side by side. A layer's parameter count is $K \times K \times C_{\text{in}} \times C_{\text{out}} + C_{\text{out}}$ , where $C_{\text{in}}$ is the number of input channels and $C_{\text{out}}$ the number of kernels. For $3\times3$ with 3 in and 64 out that is 1,792 — almost five orders of magnitude below the dense layer's 150 million.

Fewer parameters does not mean less compute, though: in practice a convolution is lowered to one large matrix multiply (the cost of matmul).

Pooling: absorbing small shifts

What the convolution gives is translation equivariance: move the cat right and the response moves right. What we ultimately want is a verdict that does not depend on position at all — translation invariance.

Pooling closes that gap. Keeping only the maximum in each 2×2 window (max pooling) halves both dimensions, and small displacements inside the window stop showing up in the output. At the same time, the lower resolution means the next layer's 3×3 window covers a wider region of the original image. That is the growth of the receptive field. Many modern designs drop pooling and let a stride-2 convolution do the shrinking instead.

Stack them and edges become objects

Stack layers and what each one responds to grows in a hierarchy. The first layer responds to changes in brightness — edges and oriented lines. It sees only a few pixels, so nothing more is available to it. The next layer does the same thing to the feature maps the first one produced: look at the "map of vertical lines" and the "map of horizontal lines" at once, and where both are strong you have a corner, where they repeat at a regular spacing you have a texture. Deeper still come parts such as an eye, then pieces of objects — a face, a car body. Nobody designed that hierarchy; it emerges when this structure is trained.

Turning scores into probabilities

At the end, the shrunken feature maps are flattened and a dense layer emits one number per class. A dense layer is fine here because its input is no longer 150,528 numbers but a far smaller summary. Those numbers are logits: raw scores whose ordering is meaningful but whose range and sum are not fixed. Softmax turns them into probabilities.

p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}

(3)

$z_i$ is the logit for class $i$ , $p_i$ its probability and $T$ the temperature (1 by default). What this says, in words, is make everything positive with an exponential, then divide so the total comes to one. The sum in the denominator is every class added up, which is exactly why the outputs always total 1; dividing by $T$ before the exponential stretches or squashes the gaps between scores. Because of that exponential, a small gap between logits becomes a large gap between probabilities.

FIG 1Lower the temperature and the distribution sharpens onto one class; raise it and it flattens. Note that the bar heights depend only on the gaps between logits — they say nothing about whether the model is actually right

The convolution in numpy

Everything above is this much code.

import numpy as np

def conv2d(x, k, stride=1, pad=0):
    x = np.pad(x, pad)                    # ring the input with zeros
    K = k.shape[0]
    H = (x.shape[0] - K) // stride + 1    # the output-size formula itself
    W = (x.shape[1] - K) // stride + 1
    out = np.zeros((H, W))
    for i in range(H):
        for j in range(W):
            win = x[i*stride:i*stride+K, j*stride:j*stride+K]
            out[i, j] = (win * k).sum()   # dot product of window and weights
    return out

(win * k).sum() is one window's dot product; the double loop is the sliding.

How this shows up on the job

Image classification becomes work when someone wants "look at a photo and sort it" automated: visual inspection on a production line, deciding what kind of document a scan is, filtering uploaded images. Every one of them starts by fine-tuning a pretrained model on your own data. Stacking convolutions from scratch almost never happens.

The knobs are largely fixed. Input resolution (224 by default, raised if you need to see fine scratches), the normalisation mean and standard deviation (use the values the pretrained weights expect), augmentation (RandomResizedCrop, horizontal flip, colour jitter), learning rate (one to two orders of magnitude below pretraining), and how much of the network to freeze. The less data you have, the deeper you freeze.

Most accidents come from a preprocessing mismatch. If resizing or normalisation differs between training and inference, accuracy quietly drops. Nothing raises an exception or a warning, so it goes unnoticed. Sharing one preprocessing function between training and serving prevents it outright.

Only augment with distortions that actually occur. A horizontal flip helps for cats and poisons text, arrows and medical images where left and right carry meaning. All the convolution gives you structurally is robustness to translation — it does not become robust to rotation or scale on its own.

Do not trust accuracy. On a line where 1% of parts are defective, answering "all good" scores 99%. Per-class precision and recall plus a confusion matrix are the minimum. Softmax probabilities are also not necessarily calibrated and tend toward overconfidence, so a rule like "auto-approve above 0.9" needs that threshold re-measured on production-like data. And when you feed in a category the model never saw in training, it will still assign high probability to one of the classes it knows. A standard classifier has no way to say "I don't know" — design your operations around that.

In design reviews the standard question is "why a convolution rather than a dense layer?" Parameter count and translation equivariance are enough of an answer. The other common one is "what does a 1×1 convolution do?", and the answer is a linear combination along channels only — a dense layer running independently at each pixel position, used to change channel counts and to build bottlenecks.

Summary

Feeding a photo into a dense layer explodes the parameter count, and moving the subject produces an unrelated input
The convolution slides one small window, buying local connectivity (fewer parameters) and weight sharing (translation equivariance) at once
Kernel size, stride and padding fix the output size; the number of kernels fixes the number of feature maps
Pooling (or a stride-2 convolution) absorbs small shifts and widens the receptive field
Stacked layers grow features from edges to textures to parts to objects
Logits become probabilities via softmax — probabilities that are not calibrated

Next: how deep this stacking can go, what breaks when it gets deep, and what fixes it.