Paper Walkthrough: OmniScientist — An AI Scientist That Actually Looks at the Raw Data

Automating every step of the research workflow doesn't help much if the only thing the agent ever sees is somebody else's summary table. OmniScientist puts raw waveforms, images and 3-D point clouds in front of the agent for the whole research lifecycle and enforces the gates in Python — walked through from the paper itself, up to the 21.7% finding buried in a seismic benchmark.

OmniScientist: An Omni-Modal Omni-Discipline AI Scientist

The junior researcher who only got the summary table

Imagine handing a junior researcher a pile of pathology slides and saying "find something." Except you don't hand over the slides — you hand over a feature table somebody else prepared: mean intensity, area, a handful of summary statistics. They will run honest statistics and write a respectable paper. But questions like "this tissue is mottled" or "the nuclear density changes only here" — the questions you can only reach by looking — were never on the candidate list to begin with.

That is exactly the gap this paper points at. Recent AI scientists have automated nearly the whole visible research workflow: ideation, code execution, manuscript preparation (§1). Yet in most of these systems, the only interface to the data is text, code, labels, or precomputed summaries. The agent inherits a human-chosen representation before inquiry begins. The paper's phrase for this is that existing systems are workflow-complete but evidence-incomplete (§1).

Which relations survive the interface?

The obvious objection is: images and waveforms can all be serialised into tokens, so what's the difference? The paper's answer is sharp — the question is not what can be serialised, but which relations survive the interface (§3.1).

A caption drops local morphology. An unordered feature vector erases temporal order. A handful of scalars hides cross-channel inconsistency. The fact that all three channels of a three-component seismometer rise at the same instant simply does not exist in "mean amplitude." Spatial, temporal, cross-channel, statistical and procedural relations are thrown away at the moment the summary is computed.

The widget below is there so you can feel that loss with your hands (an analogy for intuition, not an experiment from the paper).

Four families of scientific evidence

Rather than sorting artifacts by representation ("images", "tables"), the paper groups them by the kind of reasoning their interpretation requires (§3.1, Table 2): perceptual (images, micrographs, spectra, waveforms, 3-D structure), symbolic (documents, formulae, sequences, knowledge graphs), quantitative-statistical (tables, measurements, distributions), and procedural (trajectories, simulations, agent traces).

This taxonomy earns its keep because the same engine then runs on a seismogram, a CAD mesh, or a knowledge graph. Adding a discipline means writing one specification file — no change to the core pipeline and no domain-specific research code (§3.2). The demonstration suite covers 5 top-level discipline categories and 36 cases, ranging from 12 Feynman equations to a biomedical knowledge graph with 5 million edges. 28 of the 36 cases are perceptual; the other 8 are deliberate breadth controls, drawn from settings where looking buys you little and a text-only baseline should already be strong (§3.2).

The paper makes the contrast concrete on three cases (Figure 2): a three-component seismogram, a stained pathology tile, and a 3-D CAD model. In each, the structural cue read off the raw record is what carries the finding — whereas the same artifact delivered as a precomputed vector arrives with those relations already removed. The difference is not manuscript style; it is the space of questions you can even ask (§3.1, §5.4).

The perception layer: don't render an image first

The perception layer is what puts raw artifacts in front of the agent. The interesting design choice is that it doesn't jump straight to pictures (§4.1). It first extracts native numeric properties in the modality's own terms — FFT peaks, trend points. Visual rendering is invoked only when spatial or structural patterns are genuinely essential. And looking costs budget. The ideation-stage image budget is:

Here $B$ is how many image inspections one ideation stage is allowed, and $g$ is the number of label groups in the data. In plain terms: budget two looks per label group, then clamp that between 8 and 24. More groups buys more looking, but never without a ceiling (§5.1).

What the formula says, in words: even the simplest two-group dataset still buys 8 image inspections, and no dataset, however many groups it carries, buys more than 24.

Appendix D shows how this played out. Across the 36 runs there were 337 perception calls, 173 of them to a look_at_* tool. Signals, audio, video, 3-D structures and trajectories each expose both a native reader that returns numbers and a visual reader that renders the artifact, and the agent picks. Tables, formulae, sequences and graphs are not forced into an image at all.

Crucially, this layer is available for the entire lifecycle. All three stages can call the same perception tools (the dashed arrows in Figure 3), and Figure 4 lays out, across 16 cases, 11 modalities and all 4 evidence families, the raw observation the layer actually read next to the discovery that came out of it (§4.1).