End-to-End Driving from Scratch — Perception to Control in a Single Network

End-to-end autonomous driving learns everything from camera pixels to steering in one neural network. Starting from zero, we cover how it differs from the modular stack, imitation learning and distribution shift, why 'predicting the average' causes crashes, and why interpretability and validation remain the hard part.

End to End Learning for Self-Driving Cars

A Factory Line vs. a Single Driver

There are two broad philosophies for building self-driving software. The first is the modular stack: split the job into stages — perception (what is where), prediction (where it will move next), planning (which path we take), and control (how to move the wheel and pedals) — build each stage with a separate team, and pass data between stages in fixed formats like "list of detected objects" or "predicted trajectories." Like a factory production line, responsibilities are clear, and you can inspect exactly which station produced a defect.

The second is our topic: end-to-end driving. A single neural network takes sensor input — camera images and the like — and directly outputs a steering angle or a target trajectory. A skilled human driver doesn't consciously run stages ("detect pedestrian ahead, compute predicted trajectory..."). They see, grasp the whole situation, and their hands and feet move. End-to-end tries to reproduce that unity in a machine.

The Price of Modularity — Information Dies at the Boundaries

Division of labor has costs. First, humans decide the interface formats between stages. The moment the perception team decides "objects are represented as 3D boxes," anything that doesn't fit in a box — a pedestrian's gaze, a cyclist's wobble, cargo about to slide off a truck bed — never reaches the downstream stages. No matter how smart the planner is, it cannot use information that was thrown away.

Second, errors cascade. An object the perception stage misses does not exist for prediction or planning, and small errors at each stage get amplified downstream. Third, the objectives don't line up. The perception team optimizes detection accuracy and the prediction team optimizes trajectory error, but there is no guarantee that improving those metrics improves driving. A tiny gain in detection accuracy means little if the newly detected objects are distant parked cars irrelevant to the drive.

End-to-end attacks all three at once. Because the whole network is trained against a single loss — how good the final driving is — the network itself decides what intermediate representations to keep, and gradients (the signals that say which direction reduces error) flow uninterrupted from the output end back to the input end.

An Older Idea Than It Looks

End-to-end driving sounds like a recent fashion, but the idea is classic. In 1989, ALVINN drove a real vehicle with a small neural network mapping camera images to steering angles. In 2016, NVIDIA scaled the same framing up with a convolutional network (known as PilotNet), showing lane-keeping behavior without ever building an explicit lane detector. On the research side, UniAD — best paper at CVPR 2023 — showed a design that keeps the stages but connects them differentiably with planning as the final objective, and production automotive software has increasingly been described as moving in the end-to-end direction.

Written as math, the whole of end-to-end driving is one function.

Here $o$ is the sensor observation (e.g., a sequence of camera images), $a$ is the action (steering and acceleration, or a target trajectory a few seconds ahead), and $\pi_\theta$ is a neural network with learnable parameters $\theta$. In plain words: learn one function that takes in observations and puts out actions. The enormous problem of driving collapses, at least on paper, into this single line. The question is how to learn $\theta$.

Imitation Learning — Humans as the Teacher

The dominant approach is imitation learning, specifically behavior cloning. The mechanism is almost disappointingly simple: collect many pairs of "what was visible at that moment, $o$" and "what the human did at that moment, $a^*$" from human driving, then run supervised learning to pull the network's output toward the human's action.

Read aloud: take the difference between the network's output $\pi_\theta(o)$ and the action the human actually took $a^*$, square it, and average over all collected data. Minimize that with gradient descent. The appeal is cheap data: driving logs accumulate whenever humans simply drive, with no need to draw bounding boxes around every object by hand.

But "small error on the training data" and "behaves well in situations it has never seen" are different things — and that gap is the source of nearly every difficulty in end-to-end driving.

Errors Compound — Distribution Shift

Behavior cloning has a structural trap that ordinary supervised learning does not. If you misclassify one cat photo, the next photo is unaffected. In driving, your output changes your next input. Steer slightly too far, and the car drifts into a position a human would never occupy — an observation absent from the training data. The network's output on unfamiliar observations is even less accurate, error feeds on error, and a few seconds later you're on the shoulder. Mistakes accrue not additively but like compound interest. This is called distribution shift (covariate shift).

There are two classic countermeasures. One is DAgger: let the partially trained network actually drive, collect the off-distribution situations it gets itself into, label what a human would do there, and add those pairs to the training set — putting "the correct answer in nearly-failed situations" into the curriculum. The other is data augmentation: NVIDIA's PilotNet mounted extra cameras on the left and right of the car, synthesizing pairs of "the view when slightly off-lane" and "the correction that recovers." This patches a fundamental hole: expert demonstrations barely contain recovery maneuvers, because good drivers rarely get off track in the first place — you cannot learn recovery from watching only people who never need it.

The Average Is the Most Dangerous Answer — Multimodality

The second trap: there is often more than one correct answer. With debris ahead, swerving right is correct and swerving left is correct, and the training data contains both. Minimize the squared error above and what does the network learn? The average of right and left — driving straight into the debris. The minimizer of squared error is the conditional mean, and when the distribution of correct answers has two peaks (modes), the mean points at the valley between them — a place that is neither answer. This is the multimodality problem.

The fix, in all its variants, is to stop regressing a single point and output a distribution or a set of options instead. Three representative designs: (1) score a fixed set of candidate trajectories ("anchors") with probabilities and treat it as classification; (2) feed the navigation command (turn left / turn right / straight) as an extra input so the fork disappears — known as conditional imitation learning; (3) learn the trajectory distribution itself with a generative model such as diffusion. All of them dodge the "mean falls into the valley" failure by changing the output representation.

Behavior Cloning in Code

The skeleton is just this:

for images, expert_action in dataloader:   # human driving logs
    pred = net(images)                     # images -> trajectory / steering
    loss = mse(pred, expert_action)        # gap vs. the human's action
    loss.backward()                        # gradients flow from control back to perception
    optimizer.step()

Notice what's missing: the words "lane," "pedestrian," and "traffic light" never appear. Whether those concepts should be recognized at all is left to the network — it will grow them internally if they help reduce the loss. That is the end-to-end philosophy, and also the source of the anxiety in the next section.

Interpretability — No Answer to "Why Did It Swerve?"

With a modular stack, you can open the log of a near-miss and bisect: did perception output the pedestrian? Did prediction output the crossing? Pure end-to-end has no such seams. Between the input image and the output steering sit hundreds of millions of weights, and an answer to "why the hard swerve?" is fundamentally hard to extract. Explanation techniques like attention-map visualization exist, but nothing guarantees they reflect the true cause.

The practical compromise is end-to-end that keeps its intermediate outputs. UniAD retains detection, tracking, prediction, and planning as modules inside the network, connects them differentiably, and trains the whole thing against final planning performance. Gradients flow end to end (the end-to-end benefit), yet intermediate detections and predicted trajectories remain readable by humans (the modular benefit). The intermediate outputs serve as windows for monitoring and debugging, without the downstream stages being restricted to them alone. The BEV representation that fuses multiple cameras into a top-down map is a common shared substrate for this style of design.

Validation — "Imitates Well" and "Drives Safely" Are Different Claims

The heaviest practical issue is validation. There are two kinds of evaluation. Open-loop evaluation replays recorded drives and measures, at each moment, the displacement between the network's output trajectory and the human's. It is cheap, but it cannot reproduce the compounding effect where your output alters your next input. Closed-loop evaluation lets the network actually drive, in a simulator or a real car, and measures route-completion, violations, and interventions. Models that score well open-loop and then fall apart closed-loop are common, and it is received wisdom in the field that selecting models on open-loop metrics alone is dangerous.

A deeper problem: end-to-end sits badly with the standard safety-engineering move of decomposing requirements per stage and verifying each. You cannot write a per-stage specification like "pedestrian detection shall miss no more than X within Y meters," so assurance leans on "how many diverse situations did we drive, and what were the statistics." The situations that matter most are rare dangerous ones (the long tail), which makes simulation-based synthesis of rare scenes and the intermediate-output monitoring from the previous section the pillars of validation. This area is unresolved both technically and institutionally, and the debate between end-to-end advocates and modular holdouts continues here.

How This Is Used in Practice

At autonomous-driving and robotics companies, this knowledge isn't only for the ML engineers training the model. Evaluation-infrastructure engineers maintain separate dashboards for open-loop displacement errors (ADE/FDE — average and final displacement error) and closed-loop completion/intervention rates, and build gates so that a change which only improves open-loop numbers cannot be promoted. Driving tests in simulators such as CARLA and comparisons on nuScenes-style datasets are the usual footing. Data engineers own the pipeline that harvests intervention logs (moments when a human took back control) and feeds them into training — which in effect acts as DAgger.

The failure modes are well-worn too. First, as above, model selection by open-loop metrics: with lots of stopped-in-traffic data, a degenerate model that outputs "almost always go straight" scores a deceptively small displacement error. Second, demonstration quality: sloppy human driving gets imitated verbatim, so curating the teacher data (filtering out harsh maneuvers and violations) pays off quietly but consistently. Third, regressing trajectories with squared error without knowing about multimodality, producing the classic failure where the car drives "down the middle" at a fork. Fix the output representation — distributions or candidate selection — before reaching for a bigger model.

Summary

• End-to-end is the design philosophy of not letting humans fix the inter-stage formats, avoiding information loss at boundaries, error cascades, and misaligned objectives
• The workhorse is imitation learning (behavior cloning), which carries two structural traps: distribution shift (errors compounding) and multimodality (the mean points at the valley)
• Interpretability and validation are the biggest weaknesses; differentiable pipelines that keep intermediate outputs (UniAD-style) plus closed-loop evaluation are the current practical landing zone

For the "seeing" side that feeds all of this, see The Autonomous Driving Perception Stack from Scratch; for fusing multiple cameras into one map, see BEV Representation from Scratch. The same "observation to action in one network" framing applied to robot manipulation is the vision-language-action model — TurboVLA explained is a worked example.