Paper Walkthrough: ENEAS — Telling Real Instances From Their Doppelgangers

Ask for 'person' and you get the statues too. This paper fixes the failure where a mask is geometrically right and semantically wrong, using an embedding filter plus a conditional VLM judge — explained from first principles.

ENEAS: Embedding-guided Neural Ensemble for Adaptive Segmentation

You asked it to remove the people, and it removed the saints

Imagine you have shot a few hundred photographs of a church interior for a 3D reconstruction. Before you run the reconstruction, you want the visitors gone. So you hand a text-promptable segmentation model the word "person" and ask for masks. What comes back covers the visitors — and the life-sized statues of saints standing at the altar. The statues are stone. They are not people. But their outlines and their shading are a flawless human shape.

The paper this article covers is titled "ENEAS: Embedding-guided Neural Ensemble for Adaptive Segmentation" (Javier del Pino, Salvador Rodríguez, Alejandro Garabito, Javier Álvarez, Chema Garabito; posted to arXiv on 3 September 2026).

Its abstract makes the following claims. Text-promptable segmentation, even in the newest foundation models such as SAM 3, still fails in three ways: temporal hallucination, where the model cannot report that a target has left the field of view; spatial fragmentation, where an extreme close-up makes it segment a local texture instead of the whole object; and semantic misclassification, where visual features win over ontological reality, so that look-alike artifacts such as statues, paintings and reflections are segmented as the target entity. ENEAS gets two behaviours out of a single method: precise tracking and high-quality segmentation of one unique instance, and open-concept discovery of every instance a text query names, settled by a semantic verification layer. For tracking, the authors extend the geometrically robust SeC architecture — previously limited to point interactions — with a text-prompting adapter, and lean on its temporal memory so the target is held through disappearance without drifting to distractors and stays whole even when it fills the entire view. For discovery, the verification layer pairs fast visual embedding matching with conditional VLM refinement, invoking semantic reasoning only for ambiguous candidates, which removes the ontological errors visual-only models cannot see while keeping latency low. It was designed with 3D reconstruction in mind, where a single misclassified distractor corrupts the asset, and the code and models are public.

Geometrically correct and semantically wrong

The paper argues that these failures share one root: strong perception coupled with weak verification (§1). The ability to draw an accurate boundary is more than good enough. What is weak is the step that checks whether the thing inside that boundary truly belongs to the named concept. In the authors' own phrasing, the output is geometrically correct and semantically wrong (§1).

They ran into this while building 3D reconstruction pipelines (§1). A specific object had to be isolated across unordered, partially occluded views, and every visitor had to be removed without touching the sculptures being reconstructed. Asked for one painting, SAM 3 returned several; it masked statues as people. It was not usable for the task (§1).

In this setting the two kinds of error do not cost the same (§4.1). A false positive masks part of the very asset you are reconstructing and destroys it, whereas a false negative merely leaves a visible artifact. That asymmetry is why the design is precision-first — which cuts against the usual instinct of a detector builder, who tunes for recall.

So why can't appearance settle it? The standard modern move is to place text and images in a shared vector space and measure closeness (see CLIP from scratch). The trouble is that "a real person" and "a hyper-realistic statue of a person" sit in almost the same place in that space.

Shared parts: separating "where" from "what"

ENEAS takes a natural-language prompt $p$ and a set of frames $\{\mathbf{I}_t\}_{t=1}^{N}$ and returns binary masks (§3). The frames may be an ordered video or an unordered image collection. When the prompt names one specific instance (say, "the blue painting"), the output is one mask $\mathbf{M}_t$ per frame, and that mask is empty on frames where the object is not visible. When the prompt names a category (say, "person"), the output is a set of masks per frame, one per instance, and the count varies frame to frame (§3).

Both cases rest on the same two parts: a grounding operator $\mathcal{G}(\mathbf{I},p)$ that turns a prompt into image regions, and a segmentation operator $\mathcal{S}(\mathbf{I},\mathbf{r})$ that turns a region into a mask. In the paper's implementation these are Florence-2-Large and a Segment Anything model (SAM 2 / SAM 2.1) respectively (§3). ENEAS, in other words, is not a new detector or a new mask head. It is a paper about inserting verification between existing strong parts.

Mode 1: following one specific thing

The tracking side builds on SeC (§3.1), which augments SAM 2 with a concept-level memory of what the target is. Because it does not rely on appearance matching alone, it survives drastic viewpoint changes and targets that leave the view and come back. The catch was that SeC natively accepts only point prompts.

ENEAS adds text initialization. On a reference frame $\mathbf{I}_{t_0}$, the grounding operator locates the object the prompt describes, giving $\mathbf{r}_0=\mathcal{G}(\mathbf{I}_{t_0},p)$. Every subsequent mask comes from propagation.

In words, the mask for frame t is what the tracker draws from just two things: the single region it was given once at the start, and its memory of the frames processed so far. Nothing is re-detected each frame, so when the object is absent the mask comes back empty instead of landing on a look-alike.

Equation (1) reads plainly: the mask $\mathbf{M}_t$ for frame $t$ is what the tracker $\mathcal{T}$ draws, guided by the single region $\mathbf{r}_0$ it was given once and by $\mathcal{H}_t$, its memory of the frames processed so far. Nothing is re-detected each frame. Identity is decided once and carried forward, which is why $\mathbf{M}_t$ comes back empty rather than sliding onto a look-alike when the object is absent (§3.1).

The qualitative study on Blue Painting — an indoor sequence with viewpoint changes, occlusions and close-ups — shows the difference directly (§4.4). Grounded SAM (Grounding DINO plus SAM 2.1) drifts when the target leaves the view or is heavily occluded, firing positives on background curtains or on the subject's hair in order to satisfy the prompt. ENEAS reports a true negative — no mask — in the same frames. Under extreme close-ups, both Grounded SAM and SAM 3 fragment the object, segmenting brushstrokes or the figures painted inside it, while ENEAS keeps the painting whole (§4.4).