Paper Walkthrough: Stop Anchoring to Frame One — Scal3R's Multi-Reference Relative Pose Query

Long videos make online 3D reconstruction collapse. This paper traces the cause to extrapolation against a fixed first-frame anchor and fixes it by adding roughly 1% of trainable tokens to a frozen backbone. Explained from zero.

Scal3R: Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction

The paper we are reading

The original title is "Scal3R: Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction", by Chin-Yang Lin and colleagues at National Yang Ming Chiao Tung University and NVIDIA. It appeared at ECCV 2026 and was posted as arXiv:2609.04201 on 3 September 2026.

The abstract makes the following case. Online 3D reconstruction models perform poorly on long videos, because regressing poses relative to a fixed first-frame anchor forces extrapolation far beyond the training distribution; small drifts accumulate and amplify into significant geometric collapse. The authors observe, however, that per-frame depth stays stable throughout that failure — the backbone's local geometry remains intact, and only the global pose head breaks down. Motivated by that decoupling they introduce Scal3R, which reformulates online reconstruction as multi-reference relative pose querying. Lightweight learnable tokens amounting to roughly 1% of the parameters are injected into a completely frozen backbone via asymmetric attention, and an online pose-graph optimization system with loop closure suppresses long-range drift. Scal3R converges in 8 hours on a single GPU, reduces average ATE on KITTI by over 60% compared to the online baseline, and reaches state-of-the-art results on Virtual KITTI, Sintel, TUM-Dynamic, ScanNet and 7-Scenes.

An analogy: navigating by one landmark you can no longer see

Picture yourself walking out of a train station with a camera, covering several kilometres. Every second, from nothing but what you can see, you must state your position and heading relative to that station. That is exactly what conventional online 3D reconstruction does: models such as CUT3R and STream3R regress each frame's pose $P_t$ into a global coordinate system anchored at the first frame (§3.2).

Ten metres out, that is easy — the station is still in view. Three kilometres out it is long gone, yet the model must keep producing station-relative coordinates. Worse, existing 3D datasets cover only limited scene scales (§1), so the model has never seen numbers of that magnitude during training. It is being asked to extrapolate outside its training distribution.

And the errors compound. A few centimetres of error per frame becomes hundreds of metres over a few thousand frames, and the point cloud stretches and twists apart. The paper calls this geometric collapse and shows it happening on kilometre-scale driving sequences such as KITTI (§1, Fig. 1).

The observation: only the pose head is broken

This is the paper's starting point, and the most interesting part of it.

The authors ran an error-correlation analysis and found the failure to be strikingly localized. Global position error diverges catastrophically once the sequence leaves the training distribution, while per-frame depth prediction stays stable throughout (§1, Fig. 2b). What breaks is only the final head that converts intact local geometry into global coordinates.

That is a powerful diagnosis, because it means you do not need to rebuild the backbone — you only need to replace the pose output. And if local geometry is trustworthy, you can stop asking "where am I relative to the station" and instead ask "where am I relative to that junction I passed a moment ago". A question posed against a nearby reference has an answer inside the range the model saw during training, so no extrapolation occurs.

The paper is candid that the general insight — relative formulations generalize better than absolute ones — is already known in the visual odometry literature (§2). Scal3R's contribution is delivering it on top of a frozen 3D foundation model at almost zero additional training cost.

Mechanism 1: add tokens that only ask questions

Scal3R keeps the backbone — a 24-layer Transformer, either CUT3R or STream3R — completely frozen. Only the newly added lightweight tokens and their small heads are trained, amounting to roughly 1% of the backbone's total parameter count (§3.3).

The construction is straightforward. A pose token buffer stores the camera tokens of selected past keyframes, and a single shared base query token $\mathbf{q}$ is learned. For reference slot $k$, that frame's camera token is passed through a light MLP and added to the base token:

Here $\mathbf{q}$ is a template instruction meaning "extract the geometric relationship between the current frame and a reference", $\mathbf{c}_{r_k}$ is the camera token of the $k$-th reference frame, and $\tilde{\mathbf{q}}_k$ is the concrete question aimed at that particular reference. Equation (1) is, in plain terms, writing an address onto a generic question to produce a specific one.

The payoff is that the number of questions is free to vary. Because each token queries independently, the reference count $K$ can be changed at inference time — the paper trains with $K=3$ and defaults to $K=12$ at inference, explicitly without retraining (§3.3, §4.1).