Paper Walkthrough: Marigold V2 — Turning an Image Editor Into a State-of-the-Art Depth Estimator on One 32GB GPU

How Marigold V2 repurposes an image-editing diffusion transformer (Qwen-Image-Edit) into a state-of-the-art monocular depth estimator using QLoRA and two new losses — explained from first principles through normalization, single-step inference, iREPA, and SinkLoss.

Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation

What This Paper Claims

Guessing depth from a single photograph is a problem with no unique answer: infinitely many 3D scenes are consistent with the same 2D image. Get it right anyway and a lot opens up — single-image 3D reconstruction, synthetic bokeh, AR compositing, obstacle awareness for robots.

The paper we read here is "Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation" (arXiv:2609.08084, SIGGRAPH Asia 2026 / ACM TOG).

The abstract runs roughly as follows. Recent models still fail to generalize to out-of-distribution inputs and cannot produce sharp, detailed depth maps. So the authors revisit Marigold, a set of techniques for repurposing modern image generation and editing models built on the diffusion transformer (DiT) architecture into state-of-the-art monocular depth estimators. Their recipes target single-step inference from pretrained multi-step flow-matching models, with quantization where needed, preserving model capacity while staying cheap to run. They analyze the artifacts of naïve training and identify two effective remedies — aligning the model's internal representations with semantic features extracted from ground-truth depth, and a 2-stage fine-tuning protocol built around a novel Sinkhorn-based loss. The result is crisper, cleaner depth maps that generalize well, with a 16–26% improvement in AbsRel over the previous best on KITTI and ETH3D, resolving fur, foliage, and hair-thin edges. The same recipe reaches state of the art on other dense regression tasks such as surface normals and intrinsic image decomposition.

The Analogy: Asking a Skilled Painter to Do Surveying

Depth estimation has two lineages. One trains a regressor on sensor-captured ground truth. The other — this paper's — reassigns a model that already knows how to paint into a surveying role. That works because a generative model, in order to produce plausible images, has implicitly learned occlusion, materials, and lighting. That world knowledge comes from billion-scale data; depth annotations only exist at million scale (§2.1).

The lineage has a chronic ailment, though: boundaries blur, thin structures vanish, and projecting to a point cloud leaves flying pixels floating around silhouettes. V2 is the paper that goes after that ailment directly (§2.4).

Prep Work: Normalize Depth in the Log Domain

Ground-truth metric depth $D$ is first converted into an affine-invariant normalized log-depth target $d$ (§3.1).

Here $d_2$ and $d_{98}$ are the 2nd and 98th percentiles of $\log(D+\epsilon)$. In plain words: take the log, clip the top and bottom 2% as outliers, and stretch what remains onto $[-1,1]$. That throws away the "how many times larger" and "shifted by how many meters" ambiguity and keeps only the shape.

Why the log? In the representation comparison (Tab. 4), linear depth scores AbsRel 5.04 and disparity 5.28, while log depth wins at 4.72. The paper gives the reason as a one-liner: for small relative error $\epsilon$, $\log x_{\mathrm{pred}}-\log x_{\mathrm{gt}}\approx\epsilon$, which means an $L_1$ penalty on log depth coincides with the AbsRel metric to first order. The ruler you train against and the ruler you are measured with are now the same ruler.

Adapting the DiT: Collapsing Multi-Step Diffusion Into One Subtraction

The backbone is the image-editing model Qwen-Image-Edit-2509. Its weights are quantized to 4 bits, and only rank-128 QLoRA adapters are trained (§3.2).

The training objective is startlingly plain. The VAE encoder maps the RGB image and the normalized depth target into latents $z_I$ and $z_d$; the target velocity is defined as $v=z_I-z_d$; and the DiT $f_\theta$, conditioned on $z_I$ with the timestep fixed at $t=0.5$, is trained to regress it via $\mathcal{L}_{\mathrm{latent}}=\|f_\theta(z_I,t)-v\|_2^2$. In words: predict, in one shot, the vector that points from the RGB latent to the depth latent. Because $t$ is fixed, the rectified-flow parameterization has no trajectory left to integrate and degenerates into plain latent regression. Inference is a single pass:

Subtract the predicted vector from the RGB latent, run the VAE decoder, done. One subtraction instead of a sampling loop — that is what "cheap to run" actually means here. For background, see Flow Matching from scratch and the LoRA paper explained.

On top of latent supervision, the model is also supervised in pixel space, with an $L_1$ reconstruction loss and an $L_1$ spatial-gradient loss. The latter matches how sharply values step between neighboring pixels, which is exactly what edges are made of.