The State of 3D Generation — From NeRF to Gaussian Splatting

3D stopped being something you sculpt and became something you fit by optimization. How the representation moved from meshes to NeRF to Gaussian Splatting, how SDS turns a 2D diffusion model into a judge that supervises 3D, and what breaks when you take any of it into a game or film pipeline.

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

Not carving — fitting

Ask someone to make a 3D model and they picture Blender: grab a vertex, pull it, staple triangles together, unwrap UVs, paint a texture. That workflow has barely changed in thirty years.

What appeared in the last few years starts somewhere else entirely. Treat space itself as fog that has colour and thickness. Given a few dozen photographs of a subject, you can pose the question "how should this fog be arranged so that all of those photos are explained?" and solve it by optimization. You are not carving; you are tuning densities until the renders match the pictures. That is the road NeRF (Neural Radiance Fields) opened in 2020.

Then came the move that removes the photographs too. Text-to-image diffusion models are already good enough, so use one as the judge. Make up some rough 3D, render it from random angles, ask the model to score how much each render looks like "a red leather armchair", and push its complaints back into the 3D. That is Score Distillation Sampling (SDS).

What counts as "3D data"

• Mesh: vertices and triangles. Game engines and DCC tools are built around this
• Voxels: chop space into a grid. Simple, but memory grows with the cube of the side length
• Point cloud: points carrying position and colour. No surfaces, so it has holes
• SDF (signed distance field): a function returning distance to the surface. Smooth, but carries no colour

From a generative-model point of view, meshes have an awkward property: gradients do not flow through them. The number of triangles and the choice of which vertices connect to which are discrete structure that cannot move continuously. "Make it a bit rounder, so add one vertex" is not something a gradient can express. It simply does not fit the discipline of nudging parameters and moving in whichever direction improved things.

NeRF's answer was to stop having a shape at all. Hold the scene as a function. Feed it a coordinate $(x,y,z)$ and a viewing direction $\mathbf{d}$, and an MLP returns the colour $\mathbf{c}$ at that point and its density $\sigma$ (how much stuff is packed there). The contents are weights, so everything moves smoothly.

NeRF: integrate along a ray to make one pixel

Turning that function into an image goes like this. For each pixel, shoot one ray, call the function a few dozen times along it, and blend the returned colours front to back.

Here $\mathbf{r}(t)$ is a point along the ray, $t_n$ and $t_f$ are where the integral is cut off, $\sigma$ is density, $\mathbf{c}$ is colour, and $T(t)$ is transmittance — the fraction of light that got this far without being blocked. In plain words: add up the colours along the ray, weighting each one by how much light survived to reach it and how solid that point is. If something opaque sits in front, $T$ collapses and everything behind contributes nothing. The obvious fact that you cannot see through a wall falls out of that single product. In practice the integral becomes a sum over sample points, and once it does, the machinery is ordinary alpha compositing. This form will come back later.

Training is nothing more than rendering from the same camera positions as the photos you have and shrinking the squared error against them. The point worth pausing on is that it is not trying to generalize: it deliberately overfits to one scene. The original paper reports one to two days on a single GPU to train, and tens of seconds to render one frame. Multiresolution hash encoding (Instant-NGP) is what pulled that down to seconds-to-minutes.

One prerequisite deserves spelling out. This optimization starts from knowing where each camera was and which way it pointed. Hand it a pile of photos and, without that, there is no way to decide which direction to shoot the rays. So in practice you first estimate camera poses from the photo set with SfM (Structure from Motion — the classical technique of back-solving shooting positions from feature correspondences between images), freeze them, and only then learn colour and density. Which means every 3D reconstruction since NeRF sits on top of a separate lineage of technology: pose estimation.

That dependency is not a light one. If the estimated poses are off by a few centimetres or a few degrees, the model is being told that the same surface is in two different places, and it reconciles the contradiction by parking a faint smear of fog in mid-air. Most of the floaters you see in reconstructions are exactly this. The fussiness about capture technique later in this article comes from the same place: no amount of tuning on the rendering side buys back a broken pose estimate.