BEV Representations From Scratch — Fusing Multiple Cameras Into One Top-Down Map

How a self-driving car turns six camera feeds into a single top-down map. Starting from perspective projection, we build up to the two big design philosophies: LSS, which pushes features into 3D via a predicted depth distribution, and Transformer-style methods like BEVFormer that pull information with BEV queries.

Lift

Why a Top-Down Map at All

Picture a football broadcast. The pitch-side camera is dramatic, but it makes one thing surprisingly hard to judge: how many metres of space separate two players. That is why coaches think on a tactics board — the pitch seen from directly above. Distances, positions, and open space all read naturally on a top-down plane.

A self-driving car has exactly the same problem. It carries several cameras (six is a common rig) facing forward, backward, and to the sides, and each one captures an oblique perspective view. Yet the decisions that matter — "how far is the car in the next lane?", "does this gap fit?" — are most naturally made on a map seen from above. That representation is called BEV (Bird's-Eye View), and the machinery that fuses multiple perspective images into one BEV is our topic today.

Once you are in BEV, both object detection and path planning become 2D problems on a map, and camera-derived features live on the same grid as LiDAR-derived ones, so the two mix cleanly. Where older pipelines ran detection per camera and merged the results afterwards, the BEV philosophy is to unify the viewpoint first, then reason.

Perspective Projection — What Gets Lost When the World Becomes a Photo

First, what does "being photographed" compute? In the simplest pinhole camera model, a 3D point $(X, Y, Z)$ in camera coordinates ($Z$ pointing away from the lens) lands at image position $(u, v)$:

In plain words: divide the 3D coordinates by the depth $Z$, and you get the pixel position. Here $f_x, f_y$ are focal lengths (how zoomed-in the lens is) and $c_x, c_y$ mark the image centre; together they are the camera's intrinsics. The division by $Z$ is exactly why distant objects (large $Z$) appear small — that is perspective projection.

The crucial point is that this mapping throws information away. The moment you divide by $Z$, the depth value itself vanishes from the image. Try to invert the process and a single pixel gives you not a point but an entire ray of candidates — it could be an insect one metre away or a truck fifty metres out. Depth is not recoverable from one image alone, and this ill-posed inverse problem is the fundamental reason perspective-to-BEV conversion is hard.

The Intuition — Schools of Thought Split on "How Do You Fill In the Missing Depth?"

If depth is gone, something must stand in for it. The classical answer is IPM (Inverse Perspective Mapping): assume that everything in the image lies on the ground plane (height zero), and depth becomes unique — the intersection of the ray with the ground. Parking-assist surround-view monitors work this way. But for anything with height — cars, pedestrians — the assumption breaks, and the object smears away from the camera, more strongly the taller it is. A transform that only trusts the ground is not enough for driving scenes full of upright objects.

Deep-learning approaches therefore split into two schools:

• Push from the image side: have a network predict each pixel's depth as a probability distribution, then push the pixel's features out into 3D according to that distribution. The representative method is LSS (Lift-Splat-Shoot).
• Pull from the BEV side: lay out a grid of queries on the map first, and have each one interrogate the cameras: "what is visible at my location?" The representative family is Transformer-based, e.g. BEVFormer.

The key move in both is refusing to commit to a single depth value. When the network is confident, the distribution spikes; when it is unsure, it flattens — the same behaviour you get from softmax (the function that turns scores into a probability distribution summing to one) as you vary its temperature.