The CNN Family Tree — From AlexNet to ResNet and EfficientNet

A decade of CNNs told as two campaigns — the race for depth and the race for efficiency. ReLU, residual connections, and compound scaling explained from scratch with metaphors, interactive figures, and code.

Very Deep Convolutional Networks for Large-Scale Image Recognition

CNN history as a skyscraper race

For roughly a decade starting in 2012, computer vision was a construction race: how many layers can you stack before the building collapses? At first, eight stories was the practical limit. Then new ways of framing the structure — new architectures — made 19 stories possible, then 22, and eventually 152.

Once the height contest settled, a second campaign began: the race for efficiency — getting the same view from far less material. The image recognition running inside your phone is a product of that second act.

This article walks through the landmark CNN (convolutional neural network) architectures in order, and for each one asks the same two questions: what problem were they stuck on, and what single move broke it open? You don't need to memorize names. Remember the problem–solution pairs, and ten years of progress reads like one continuous story.

Prerequisite: convolution in thirty seconds

The basic building block of a CNN is the convolutional layer. A small filter — say, a 3×3 grid of numbers — slides across the image, and at each position we measure how well the filter matches the pixels underneath, producing a feature map. Early layers respond to simple patterns like edges and color gradients; deeper layers combine those into complex parts like "an eye" or "a wheel." We covered this carefully in Image classification basics, so start there if this is new.

The one intuition to carry forward: deeper networks can express more complex concepts by composing simpler parts. That's why everyone wanted depth. The problem was that stacking layers naively broke training.

AlexNet (2012) — the eight-story starting gun

The story begins at the 2012 ImageNet competition (ILSVRC). AlexNet — five convolutional layers plus three fully connected ones, eight in total — won by a margin of more than ten points (15.3% top-5 error against the runner-up's 26.2%), in a field where a good year's progress had been one or two points. The deep learning boom starts here.

In hindsight, AlexNet's decisive moves were three:

1. It used ReLU as the activation function. The sigmoids and tanhs that dominated before flatten out at both ends, so the gradient — the learning signal — shrinks toward zero. ReLU has a constant slope of 1 on the positive side, so the signal survives all the way down through deep networks.

2. It trained on GPUs. Convolutions are stacks of matrix arithmetic, a perfect match for gaming GPUs. Memory was so tight at the time that the network had to be split across two cards — a brute-force hack that worked.

3. It fought overfitting with Dropout. Randomly disabling neurons during training prevents the network from leaning on any single pathway. That's how a model with roughly 60 million parameters — enormous for its day — avoided becoming a memorization machine.

VGG (2014) — the aesthetics of "3×3 only"

AlexNet's filters were an artisanal mix of 11×11 and 5×5. In 2014, VGG simplified radically: use only 3×3 filters, and stack them deep.

Why is 3×3 enough? Stack two 3×3 convolutions and each output pixel has "seen" a 5×5 region of the input; stack three and it's 7×7. Small filters, composed, give you the same field of view (receptive field) as a large one. And it's a bargain: with $C$ channels, one 7×7 filter costs $49C^2$ parameters, while three 3×3 filters cost $27C^2$. Same receptive field, fewer parameters, and more nonlinearity (three ReLUs instead of one).