2026-08-13·FREE·9 min read

The Autonomous Driving Perception Stack from Scratch — What Cameras and LiDAR Each Bring to the Table

How a self-driving car figures out what is around it, starting from sensor physics. We cover the complementary strengths of cameras and LiDAR, the bird's-eye-view grid where all sensors meet, occupancy grids grown by Bayesian updates, and the tug-of-war between safety requirements and compute budgets — no prior knowledge assumed.

#autonomous-driving #lidar #camera #sensor-fusion #occupancy-grid #detection

ModalityimageTaskdetection

How a Car "Sees" the World

When you walk down a dark street, you don't rely on your eyes alone. Your feet feel the bumps in the pavement, your ears catch the bicycle behind you, and your brain merges it all into a single map of your surroundings. A self-driving car does essentially the same thing: it measures the world with several sensors of very different character, reconciles their disagreements into one map, and on top of that map estimates what is where and what it will do next. This chain of processing is called the perception stack.

The stack roughly flows as: sensor input → object detection and segmentation → tracking → motion prediction, and its output feeds the planner, which decides how to drive. This article covers the front half — the "seeing" part — meaning the sensors and the representations built from them. There is also a rival school of thought that throws out this division of labor entirely and trains one network straight from sensors to steering; that route is the subject of End-to-End Driving from Scratch.

Cameras vs. LiDAR: Physics Decides Who Is Good at What

Start with what each sensor fundamentally is. Skip this, and every downstream design choice looks arbitrary.

A camera is a passive sensor: it collects sunlight or streetlight bouncing off objects. Its pixels are dense and carry color and texture, so it excels at telling what things are — "that's a stop sign," "that's a lane marking." It is also cheap, so you can mount many. Its weakness: a single image contains no direct depth. Projecting the 3D world onto a 2D sensor destroys distance information. It also struggles with glare, darkness, and water droplets on the lens.

LiDAR is an active sensor: it fires its own laser pulses and measures how long the reflection takes to return, which yields distance directly. Range measurement is direct and accurate, and it works in total darkness. In exchange, what you get is a sparse point cloud with no color, and distant objects are hit by fewer and fewer points. Rain droplets, fog, and snow scatter the laser, so performance degrades in bad weather, and the hardware has historically been on the expensive side.

In other words, their weaknesses don't overlap. The camera is strong on "what" and weak on "where"; LiDAR is strong on "where" and weak on "what." This complementarity is exactly why autonomous vehicles carry both. Most vehicles add radar on top — radio waves punch through rain and fog, and the Doppler effect gives relative velocity directly — for a third eye with yet another failure mode.

The Camera's Fundamental Problem, in One Equation

Let's turn "cameras don't know depth" into a formula. Under the pinhole camera model, back-projecting a pixel into 3D looks like this:

\mathbf{X} = d \, K^{-1} \tilde{\mathbf{u}}

(1)

Here $\tilde{\mathbf{u}}$ is the pixel position (in homogeneous coordinates), $K$ is the camera intrinsics matrix (focal length, image center, and so on packed into one matrix), $d$ is the depth at that pixel, and $\mathbf{X}$ is the recovered 3D point. In plain words: a pixel only tells you "the object lies somewhere along this ray" — to pin down where on the ray, you need the depth $d$ from somewhere else. LiDAR physically measures that $d$ for you. If you want to drive on cameras alone, a neural network has to estimate it. Much of the architectural debate in perception boils down to who pays for $d$ .

BEV: Merging Every Sensor into One Top-Down Map

Each sensor lives in its own coordinate system — the camera in its image plane, LiDAR in 3D point space — and downstream tracking and planning can't work with that patchwork. So modern stacks convert everything into a BEV (Bird's Eye View) grid, a lattice parallel to the ground. Picture graph paper with your car at the center, seen from directly above.

BEV makes a good meeting point for three reasons. First, objects don't occlude each other: in an image the near car hides the far car, but from above, everyone stands on their own cell. Second, scale is uniform: in an image a distant car shrinks, but in BEV a car 5 m away and a car 50 m away are drawn the same size, so distances on the grid are real distances. Third, the planner wants exactly this top-down map, so the output of perception is directly the input of planning.

LiDAR points already carry 3D coordinates, so collapsing the height axis drops them onto the BEV grid almost for free. The hard part is the camera: as equation (1) says, you must estimate depth while "lifting" pixels into 3D. The design space of that lifting step (pushing pixels out along a predicted depth distribution, or querying from the BEV side with a Transformer) is a big topic of its own, covered in BEV Representations From Scratch.

Occupancy Grids: "I Don't Know What It Is, But Something Is There"

The most classic and robust representation living on that grid is the occupancy grid: each cell holds the probability that it is physically occupied, and every incoming sensor observation grows that belief via a Bayesian update. In practice you store not the probability itself but its log-odds:

l_t = l_{t-1} + \log\frac{p(\mathrm{occ}\mid z_t)}{1 - p(\mathrm{occ}\mid z_t)} - l_0

(2)

Here $l_t$ is the cell's log-odds (its "occupiedness") at time $t$ , $z_t$ is the observation that just arrived, $p(\mathrm{occ}\mid z_t)$ is the sensor model — the probability the cell is occupied given this observation — and $l_0$ is the prior before any observation. In plain words: each new observation simply adds to the cell's score (a laser return: evidence of occupancy) or subtracts from it (a laser passing through: evidence of free space). Bayes' rule is full of multiplications, but taking the logarithm turns it all into addition — that's the trick.

The value of an occupancy grid is that the car can stop safely without recognizing what the obstacle is. A detector can only draw boxes around classes it was trained on (cars, pedestrians, cyclists…). A sofa dropped on the highway, a fallen tree, an overturned truck — a detector may miss objects absent from its training data, but the occupancy grid fills in from the raw physical fact: "the laser bounced back here, therefore something is here." Classification-based detection plus classification-free occupancy: this two-layer defense is the backbone of safety design, and lately there is an active line of work predicting the occupancy grid itself with neural networks ("occupancy networks").

The core of the update fits in ten lines of code:

import numpy as np

L_OCC, L_FREE = 0.85, -0.4   # log-odds increments from the sensor model

def update_grid(logodds, hits, passes):
    """hits: cells where the laser returned; passes: cells it flew through (bool grids)"""
    logodds[hits]   += L_OCC     # push toward "occupied"
    logodds[passes] += L_FREE    # push toward "free"
    return np.clip(logodds, -10, 10)   # prevent saturation the cell can never recover from

prob = 1 / (1 + np.exp(-logodds))      # back to probability: just a sigmoid

When to Fuse: Early or Late

There is also a spectrum of choices for where camera and LiDAR meet. Late fusion lets each sensor run all the way to detections independently, then reconciles the resulting box lists. The separation is clean — if one sensor dies, the other's pipeline survives — and it is easier to validate. Early fusion mixes near-raw features instead. It can catch objects that only emerge when weak evidence is combined — "faintly visible in the camera, only a few LiDAR returns" — but it inherits full sensitivity to time-synchronization and calibration errors between sensors. BEV serves as the natural meeting point for this early fusion, too.

Safety Requirements and Compute Budgets: Intelligence with a Deadline

Perception carries two constraints most AI systems never face: a deadline and an assumption of failure.

At 60 km/h, a car covers about 17 meters per second. Within one LiDAR sweep or one camera frame interval, the stack must finish detection through prediction — every single cycle, no exceptions — and not on a cloud GPU but within the power and thermal envelope of an embedded computer in the trunk. That makes the computational order of your representation an existential question. Store 3D space directly as voxels (little cubes), and the cell count explodes cubically as you raise the resolution per axis. Collapsing the height axis into a 2D BEV grid is, among other things, a decision to trade one order of growth for making the deadline.

FIG 1How cost grows as the grid side length n increases — a 2D grid grows as n², a 3D voxel grid as n³. Switch to the log scale and the difference in exponents becomes a difference in orders of magnitude. Collapsing the height axis in BEV buys back exactly that one exponent to meet the deadline

The second issue is how to treat confidence. A neural network's classification output comes out of a softmax and looks like a probability — but looking like a probability is no guarantee that it matches how often the model is actually right. A detector that says "pedestrian, 99%" while being correct only 90% of the time leads straight to an accident the moment the planner reasons "99% means no evasive action needed." Aligning reported confidence with empirical accuracy is called calibration, and the standard first move is temperature scaling — adjusting a single temperature parameter inside the softmax.

FIG 2Raise the temperature and the distribution flattens (less confident); lower it and it sharpens (more assertive). An overconfident detector's outputs are re-aligned with their true hit rate by raising the temperature — that is calibration by temperature scaling

How This Is Used in the Field

What a perception engineer on an autonomous-driving or logistics-robot team touches day to day is less the model itself than everything around it.

Data and evaluation: compare methods on public datasets like nuScenes and KITTI, then re-evaluate on internal data. For BEV detection, look beyond average precision at IoU thresholds — always check recall broken down by distance band (0–30 m / 30–50 m / …), because long-range misses vanish into the average.
Parameters you actually turn: the point-cloud preprocessing voxel_size (coarser is faster but small objects disappear), the detection confidence threshold and nms_iou_threshold for duplicate suppression, and the occupancy grid's cell resolution plus the sensor model's l_occ/l_free. Standard tooling: replay sensor streams with ROS 2 bag files, visualize with RViz-family tools, and deploy on-vehicle by exporting through ONNX to TensorRT with FP16/INT8 quantization to fit the deadline.
Pitfall #1: extrinsic calibration drift. The relative pose between camera and LiDAR (the extrinsics) is calibrated at the factory, but vibration and minor impacts shift it by degrees over time. Early fusion is fragile to this and fails silently — LiDAR points get painted with the colors of the neighboring car in the image. You need periodic recalibration or online self-calibration.
Pitfall #2: time synchronization. A few tens of milliseconds of timestamp skew between camera and LiDAR places a fast-moving object nearly a meter away from where it really is. Verify hardware triggering or PTP sync first. When "fusion accuracy looks bad," the iron rule is: suspect the clocks before the model.
Pitfall #3: the gap between evaluation and the road. Accuracy measured on data collected on sunny afternoons does not hold at a rainy tunnel exit at night. Set thresholds without stratifying evaluation by weather, time of day, and region, and you will ship a detector that looks fine on average while systematically missing in specific conditions.

Takeaways

Cameras answer "what," LiDAR answers "where." Their weaknesses don't overlap, which is why cars carry both
All sensors converge on the BEV grid — no occlusion, uniform scale, and direct hand-off to planning
The occupancy grid is the classification-free safety net: the car can stop without knowing what the obstacle is
Deadlines and failure assumptions dominate the design. Dropping one exponent via BEV and calibrating confidence are both consequences of that

The internals of the detector itself are covered in Object Detection from Scratch, and the ways of lifting camera images into BEV in BEV Representations From Scratch.