End-to-End Driving from Scratch — Perception to Control in a Single Network

End-to-end autonomous driving learns everything from camera pixels to steering in one neural network. Starting from zero, we cover how it differs from the modular stack, imitation learning and distribution shift, why 'predicting the average' causes crashes, and why interpretability and validation remain the hard part.

End to End Learning for Self-Driving Cars

A Factory Line vs. a Single Driver

There are two broad philosophies for building self-driving software. The first is the modular stack: split the job into stages — perception (what is where), prediction (where it will move next), planning (which path we take), and control (how to move the wheel and pedals) — build each stage with a separate team, and pass data between stages in fixed formats like "list of detected objects" or "predicted trajectories." Like a factory production line, responsibilities are clear, and you can inspect exactly which station produced a defect.

The second is our topic: end-to-end driving. A single neural network takes sensor input — camera images and the like — and directly outputs a steering angle or a target trajectory. A skilled human driver doesn't consciously run stages ("detect pedestrian ahead, compute predicted trajectory..."). They see, grasp the whole situation, and their hands and feet move. End-to-end tries to reproduce that unity in a machine.

The Price of Modularity — Information Dies at the Boundaries

Division of labor has costs. First, humans decide the interface formats between stages. The moment the perception team decides "objects are represented as 3D boxes," anything that doesn't fit in a box — a pedestrian's gaze, a cyclist's wobble, cargo about to slide off a truck bed — never reaches the downstream stages. No matter how smart the planner is, it cannot use information that was thrown away.

Second, errors cascade. An object the perception stage misses does not exist for prediction or planning, and small errors at each stage get amplified downstream. Third, the objectives don't line up. The perception team optimizes detection accuracy and the prediction team optimizes trajectory error, but there is no guarantee that improving those metrics improves driving. A tiny gain in detection accuracy means little if the newly detected objects are distant parked cars irrelevant to the drive.

End-to-end attacks all three at once. Because the whole network is trained against a single loss — how good the final driving is — the network itself decides what intermediate representations to keep, and gradients (the signals that say which direction reduces error) flow uninterrupted from the output end back to the input end.

An Older Idea Than It Looks

End-to-end driving sounds like a recent fashion, but the idea is classic. In 1989, ALVINN drove a real vehicle with a small neural network mapping camera images to steering angles. In 2016, NVIDIA scaled the same framing up with a convolutional network (known as PilotNet), showing lane-keeping behavior without ever building an explicit lane detector. On the research side, UniAD — best paper at CVPR 2023 — showed a design that keeps the stages but connects them differentiably with planning as the final objective, and production automotive software has increasingly been described as moving in the end-to-end direction.

Written as math, the whole of end-to-end driving is one function.

Here $o$ is the sensor observation (e.g., a sequence of camera images), $a$ is the action (steering and acceleration, or a target trajectory a few seconds ahead), and $\pi_\theta$ is a neural network with learnable parameters $\theta$. In plain words: learn one function that takes in observations and puts out actions. The enormous problem of driving collapses, at least on paper, into this single line. The question is how to learn $\theta$.

Imitation Learning — Humans as the Teacher

The dominant approach is imitation learning, specifically behavior cloning. The mechanism is almost disappointingly simple: collect many pairs of "what was visible at that moment, $o$" and "what the human did at that moment, $a^*$" from human driving, then run supervised learning to pull the network's output toward the human's action.

Read aloud: take the difference between the network's output $\pi_\theta(o)$ and the action the human actually took $a^*$, square it, and average over all collected data. Minimize that with gradient descent. The appeal is cheap data: driving logs accumulate whenever humans simply drive, with no need to draw bounding boxes around every object by hand.

But "small error on the training data" and "behaves well in situations it has never seen" are different things — and that gap is the source of nearly every difficulty in end-to-end driving.