Feed an image to an ordinary neural network and it falls apart at once. A 100×100 image means 10,000 inputs. Make the next layer 10,000 units too, and you need 100 million weights. And shift the image by a single pixel, and the network treats it as an entirely different input.

What solved both problems at the same time was convolution. The idea started out in the biology of the 1960s. Hubel and Wiesel discovered that cells in a cat's visual cortex respond only to a narrow patch of the visual field, and that they respond strongly to lines at one particular orientation.

Kunihiko Fukushima built a model of this in the neocognitron, published in 1980. Then in 1998 Yann LeCun combined it with backpropagation and put it to practical work as LeNet, reading postal codes.

What is convolution actually doing

Take a small window — 3×3, say — and slide it across the image starting from the top left. Multiply each of the nine pixels inside the window by a weight and add the results together. That result goes out as a new image. The combination of weights in the window is called a kernel, or a filter.

Three properties of this arrangement are decisively important.

Stack convolutional layers, and the shallow ones come to capture simple features such as lines and corners, while the deep ones capture more complex shapes and patterns. Inserting a shrinking operation called pooling in between lowers the resolution you work at stage by stage, while widening the region that a single output is looking at — its receptive field.