Paper Walkthrough: SceneMosaic — Physically Valid 3D Rooms from One Image, in Minutes, in Many Variations

How to turn a single picture of a room into a simulation-ready 3D scene with zero collisions, in minutes rather than hours, and with several coherent furniture rearrangements thrown in. A walkthrough of how SceneMosaic stitches the speed of image-to-3D models to the precision of VLM agents, with the equations and interactive figures.

SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution

In one sentence

Let an image model sketch the first draft, let AI agents fix it — but split the room into small tiles first, so that combining the fixed tiles yields many rooms instead of one.

The paper is "SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution" (arXiv:2609.05594, The University of Hong Kong and others, September 2026). Use that title when you search for or cite it.

Here is the abstract in plain terms. Robot learning and games need large numbers of diverse indoor scenes that a simulator can load, and generating them at scale is hard. Agentic text-to-3D pipelines built on vision-language models (VLMs) produce high-fidelity scenes but pay for it with costly place-and-refine iterations on every object. Parametric image-to-3D models are fast thanks to priors learned from 2D images, but their object poses are imprecise and often physically invalid. Worse, both families produce a single layout per input, so they cannot reflect how real rooms are constantly rearranged by the people who live in them. SceneMosaic combines the two: it takes an initial candidate from the image-based prior and then evolves it with VLM agents, getting efficiency and physical validity at once. During evolution it exploits the locality of natural scenes, decomposing a scene into independent local units that evolve separately and are recomposed via a Cartesian product. On SceneEval-100 the method matches the strongest agentic baseline in semantic layout quality with a 24× speedup, substantially reduces physical violations, and receives the highest human ratings.

The metaphor: rearrange a room corner by corner, not all at once

Try to rearrange a whole room in your head and you stall: move the sofa and the TV faces the wrong way, move the TV and the shelf is in the way. Real people think in corners — the desk area, the TV area — because what sits on the desk has almost nothing to do with which way the bed faces.

That is the heart of SceneMosaic. Split the room into groups of "a supporting object (desk, floor, wall) plus whatever rests on it", tidy each group independently, and then combine. Three options for the desk corner and two for the TV corner give six rooms for free. The name comes from exactly this: a big picture assembled from small tiles.

What was wrong: speed or correctness, pick one (§1)

The paper sorts prior work into two camps. Agentic text-to-3D has a VLM loop "place → look → fix"; quality is high, but state-of-the-art systems such as SceneSmith and SAGE need hours per scene. Parametric image-to-3D estimates every object's mesh and pose from one picture in a single pass; it is fast and inherits the realistic feel of the photo, but poses drift, so objects interpenetrate or float (§1, §2).

The second complaint is the lack of diversity. Both camps give one layout per input, and rerunning with a different seed rarely yields the kind of structured rearrangement that keeps the room's function intact while moving things around (§1). For a robot learning in simulation, seeing the same room in many states is what generalization feeds on, so this is a practical hole.

The pipeline: image → objects → tree → local units (§3.1–3.2)

The input is one reference signal (an image, or a prompt turned into an image); the output is one base scene plus $m$ variants. Each object $o_i$ carries a mesh $\mathcal{M}_i$ and a layout tuple $\mathbf{T}_i$ of position $\mathbf{p}_i$, rotation $\mathbf{R}_i$ and scale $\mathbf{s}_i$ (§3.1).

1. Object registration: a perception agent inventories the objects in the image, SAM3 segments each one, and an iterative check repairs missing instances, duplicate masks and ambiguous boundaries (§3.2).
2. Per-object 3D: SAM3D reconstructs every object independently, giving a mesh and an initial pose. This is the "image-based prior" and the source of the speed.
3. Relation reasoning: a manager agent dispatches support, containment, wall recovery and similar subtasks to specialist agents, ordered by a DAG of dependencies (§3.2).
4. Scene tree and local units: attach/contain relations form a hierarchy, and a local unit is a non-leaf node (the anchor) together with its immediate children (§3.2).

Units are only mostly independent. A few relations cross unit boundaries — "floor-standing chairs face the desk", "desk-mounted monitors share the desk's orientation" — and these are extracted as cross-unit constraints and injected into each unit's prompts (§3.2, Appendix §8). Constraints are generated only for strong functional relations with a unique reasonable configuration; weak pairs such as a painting and a sofa are deliberately excluded, because over-constraining kills diversity (Appendix §8).

Before going further, feel why this decomposition is the source of diversity. With $n$ units and a handful of variants each, the number of candidate scenes grows as the variant count to the power $n$.