Paper Explained: Holi-World — Turning First-Person Recordings into a Ledger of the Room, Then Teaching a VLM to Reason over It

Holi-World (arXiv:2610.26549) automatically turns first-person recordings into a record of room geometry, furniture, and the trajectories of objects people pick up, then converts that record into about 80K spatial and temporal QA pairs to train a 4B vision-language model. A from-scratch walkthrough of the pipeline, the numbers, and the limits.

Holi-World: Building Holistic World Intelligence from Human-Ego Experience

What problem this paper is solving

The paper is "Holi-World: Building Holistic World Intelligence from Human-Ego Experience" by Hao Li, Xiyuan Zheng, Yuanyuan Gao, and colleagues at Ropedia, NTU, HKUST, and PKU (arXiv:2610.26549, October 19, 2026). The first page points to the code on GitHub (Ropedia/holi-world) and to the model and dataset on Hugging Face under ropedia-ai.

Here is the abstract in plain terms. Video recorded from a person's own point of view — egocentric video — is a rich source for learning how the world is laid out and how it changes when people interact with it. To use it at scale, though, you have to turn an ever-growing pile of recordings into training material that ties together scene structure, what objects are, and the history of interactions, across viewpoints and over time, without leaning on expensive manual labeling. The authors propose HOLI-WORLD, a connected ecosystem covering capture, world representation, automatic data construction, model training, and independent evaluation. Its core, HoliWorld-Foundry, combines static and dynamic perception to automatically build "world records" that link the persistent structure of a scene and the changing states of objects to interaction histories and the observations that support them. HoliWorld-Data organizes records from about 100 self-collected scenes into learning resources, including roughly 80K question–answer pairs used to train HoliWorld-VL-4B. Separate human annotations of held-out recordings form HoliWorld-Bench, which measures both how accurate the automatic annotations are and what the trained model has learned. The authors report strong static detection and dynamic tracking on ScanNet, ScanNet++, Aria Digital Twin, and HoliWorld-Bench, and large gains in spatial and temporal reasoning on HoliWorld-Bench and EgoDynamic4D after training on HoliWorld-Data.

Since this sits in our time-series section, it is worth being clear about what the "series" is. It is not a column of numbers like a stock price. It is where the same object was at each moment. Once you can build that sequence, you can ask questions such as "when did the cup start moving?" and "where did it end up?" — questions about order, timing, and final state that a single frame cannot answer.

An analogy: a ledger of the house, not a diary

A first-person recording is like a diary of someone's day. They walked into the kitchen, picked up a mug, carried it to the living room, and set it on the coffee table. A diary lists events in order, which is fine for retelling the day but awkward for lookup. If you want to know where the mug is now, or whether the shelf is to the left or the right of the bed, you have to reread the whole thing and reconstruct the answer in your head.

Holi-World reads the diary and fills in a ledger of the house instead. Each piece of furniture that stays put gets one row with its position and size. Each object that a hand moves — the mug — gets a column of positions, one per moment. Every row carries a note saying which frames it was confirmed in. With a ledger like that, writing a quiz becomes mechanical. "What is behind the bed?" is answered from the furniture rows. "Where was the cup placed?" is answered from the last entry in the mug's column. And because every answer points back to frames, you can check it.

What the paper contributes is not just a way to represent the ledger but the factory that fills it in without human labor (the Foundry), plus a separate, human-annotated check of how accurate that ledger is (the Bench), wired together with model training into one loop (§1). That last point matters: if the same automatic system both writes the answers and grades them, its mistakes become "correct" by definition.

The big picture, and the groundwork

The system has five parts (§3.1, Figure 2).

An IMU measures acceleration and rotation rate, which helps work out how the camera moved between frames.

Before looking for any objects, the Foundry pins down the shape of the room and where the camera was (§3.2.1). Visual-inertial SLAM — a method that estimates the camera's own position and a map of the surroundings at the same time — recovers camera poses and a sparse set of 3D points from the stereo video and IMU. Depth Anything 3 (DA3) then estimates depth for each keyframe. From these, the system fits a 3D Gaussian Splatting model of the scene, which represents it as a large collection of small colored ellipsoids, renders refined depth back at the observed camera poses, and fills in regions where things were moving with per-frame DA3 depth. The result, for every moment $t$, is four things: an image $I_t$, a depth map $D_t$, the camera's internal parameters $K_t$ (focal length and the like), and a transform $T_t$ that maps camera coordinates into room coordinates.