Use nothing but small 3×3 kernels and simply stack them. Two stacked 3×3 kernels cover the same receptive field as one 5×5 kernel while using fewer weights and adding more nonlinearity. The structure is simple and easy to reason about, which is why it is still used as a baseline for comparison today.

Apply kernels of different sizes in parallel and concatenate the results. It also introduced the "bottleneck" structure, which uses 1×1 convolutions to cut the channel count before the expensive operations, holding the compute down.

Introduced the "residual connection," which skips over layers and adds the input straight back in. This made it possible to train 152 layers, an absurd depth at the time.