Lucida, explained — turning a room video back into parts you can move, by deferring precision to the end

A system that rebuilds a real indoor scene from video as individually movable 3D assets. It keeps the parse–generate–place order but redistributes what each step is allowed to demand, and hands the final placement to a VLM policy that operates a 3D editor in a closed loop.

Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling

Turning a room back into parts you can move

You walk around your room with a phone. From that video, the desk, the chairs and the mug come back as 3D models you can each grab and move independently. That is the form you want if you are building a training environment for a robot, or a stage for AR/VR.

The paper behind this article is titled "Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling" (arXiv:2608.30821, ByteDance Seed / Peking University / Zhejiang University, 31 August 2026).

Here is what the paper claims, up front. Composable scene modeling aims to recover a real indoor scene as complete, editable object assets arranged as observed, giving robot simulation and embodied AI a simulation-ready replica whose objects can be manipulated individually. Existing pipelines decompose the task into three steps — parse the observations into instances, generate an asset for each, place each asset back — but every step presumes an input a cluttered capture rarely provides: accurate instance geometry, unoccluded views, and assets that accurately match the observations. Lucida keeps this order but redistributes the requirements, so each step consumes only what a real capture reliably provides, and precision is reached at the end of the pipeline rather than demanded at its start. It parses the video into a scene graph whose nodes carry per-instance multi-view evidence, generates a complete asset for each instance from that evidence, and places assets with GizmoAct — a VLM policy that casts placement as multi-turn GUI interaction, manipulating the object's gizmo in a closed loop and deciding for itself when alignment is reached. Across scene-level 3D object detection, object pose estimation and scene reconstruction, the paper reports a 69% mAP improvement over Boxer on R2S-Scene, ADD-SB@0.05 raised from 57.8% to 83.4% on CA-1M, and scene F-Score raised from 0.794 for SAM 3D to 0.924.

The analogy: a modeler is not accurate at the start either

The paper's own analogy is the clearest way in (§1).

A human modeler handed the same footage satisfies none of those requirements. They review the video to note what is present, build or retrieve a model for each object, drop every model into the scene, and nudge it against the observation — switching viewpoints, correcting the residual — until the two agree. In the paper's words, precision is reached at the end of that process, not demanded at its start.

The name comes from the camera lucida, the drawing aid that superimposes the real scene onto the artist's page: you see your own line on top of reality and fix the gap.

Where the three-step pipeline actually jams

The division of labour into parse → generate → place is fine in itself; strong methods exist for each. The trouble is the preconditions (§1).

• Parsing is expected to deliver accurate instance-level estimates — precise masks, clean per-instance point clouds, or a metric 3D layout. But occlusion leaves object boundaries ambiguous, and four near-identical chairs make cross-view association and de-duplication unreliable.
• Generation presumes an unoccluded, centred view of the object, or a cleanly segmented point cloud of it. Indoor footage readily yields neither.
• Placement, typically cast as 6/9-DoF pose estimation, assumes an asset that agrees geometrically with a segmented observation — while generated assets deviate from the real object and depth is noisy.

And because the three steps run in a fixed order, an unmet requirement at any step degrades every step after it. Going end-to-end does not escape this: the paper cites evidence that predicting an object's model and pose jointly degrades both, and that generating all the assets of a scene with a single model is limited by scarce paired training data and generalizes poorly.

Lucida's move: ask each step only for what a capture reliably gives

This is the paper's central design decision (§1, §2). The order stays; the requirements move.

• Parsing no longer has to output a precise instance reconstruction. It discovers object instances, consolidates their multi-view evidence, and produces a representative 3D estimate good enough to initialize what follows.
• Generation conditions on that per-object evidence bundle rather than a clean crop or a segmented point cloud, completing structure and appearance under occlusion. The paper positions this as newly practical thanks to image generation and editing with unified multimodal models.
• Placement accepts an asset that need not match the observation exactly, and recovers the instance's 9-DoF pose in the point cloud from a coarse initial state and a referring cue.

In one line: precision is bought back by the closed loop at the end.

Parse: turning video into an object-centric scene graph (§2.1)

The input is posed RGB(-D) observations $\mathcal{I}=\{(I_i, D_i, K_i, T_i)\}_{i=1}^{N}$, where $I_i$ is an RGB image, $D_i$ a depth map, $K_i$ the camera intrinsics and $T_i$ the camera pose. From these, Lucida builds a scene graph $G=(V,E)$. Each object node $o$ carries an evidence bundle:

Read it as "the dossier collected on this object". $\mathcal{V}_o$ holds the multi-view observations in which it appears, $\mathcal{M}_o$ the associated masks or boxes, $\mathcal{P}_o$ the partial point-cloud observations, $b_o$ a representative 3D box, and $c_o$ a category name or referring description (something like "the blue chair by the window"). Edges in $E$ record spatial relations such as support, containment and adjacency.

The bundle serves both later steps: generation uses it as context to synthesize an occluder-free image, and placement takes its coarse initial state from it.

Parsing runs in three stages.

Geometry-aware keyframe selection and object discovery. Adjacent frames overlap heavily, so running object discovery on every frame is redundant. For each frame pair $i,j$, a similarity $s(i,j)$ is computed from covisibility and temporal separation. Covisibility is the fraction of sampled 3D points from frame $i$ whose projected depth in frame $j$ agrees with $j$'s observed depth within $\delta$; the temporal term damps the influence of distant frames so keyframes stay spread over a long sequence. Traversing in temporal order, a frame is retained when its similarity to every already-selected keyframe falls below a threshold. On the selected keyframes a VLM identifies object instances and a 3D detector predicts a box for each; observations are then grouped across keyframes by semantic and geometric consistency in the common 3D frame.

Object-centric full-sequence evidence consolidation. Keyframes are chosen by scene-level covisibility, so they do not guarantee sufficient observations for every individual object. For each object, a representative 3D box is selected from its geometrically consistent boxes across keyframes. If the keyframe boxes are not consistent enough, the object is propagated to nearby frames with video segmentation and tracking, the extra observations are lifted into 3D, and the selection is repeated on the expanded evidence. The representative box is then projected into the input frames, the projected 2D regions serve as prompts for per-frame 3D box estimation, and the results are validated against the representative box and observed geometry — only the consistent ones survive as full-sequence evidence.

Relation-aware scene refinement. Processing objects independently leaves scene-level errors: wrong grouping, spatial inconsistency. Appearance and geometry are compared across objects to correct erroneous merges and splits, and spatial relations are inferred. For a resting object with no valid support in the current graph, additional views are searched for the missing supporting object, which is added only after multi-view geometric validation.

Generate: fill occlusions in image space, not in 3D (§2.2)

This stage turns each evidence bundle into a complete, standalone asset.

The obvious move would be to complete the incomplete point cloud in 3D. The paper declines. In real captures, occlusion and noisy depth leave per-object point clouds incomplete and unreliable for direct 3D completion, whereas multi-view RGB observations reveal complementary parts of the object across viewpoints. So Lucida first synthesizes a complete object-centric image from the visual evidence, then lifts it to 3D.

Concretely: a small set of reliable, complementary views is selected from $\mathcal{E}_o$, favouring clear object visibility and geometric agreement with the representative 3D box while encouraging diversity in viewpoint and occlusion. Set-of-Mark prompting marks the target in each view; a VLM organizes the views into an anchor plus references and writes an editing instruction for amodal completion. An image-editing model produces a complete, isolated object image, and an image-to-3D model converts it into the asset $A_o$, coarsely initialized in the scene using $b_o$. Position, orientation and anisotropic scale are left to the next stage.