Paper Walkthrough: ToolArtist — Search, Draw, or Redraw? The Image Agent That Decides for Itself

Drawing "a harvester collecting dragon's blood resin on Socotra in 1955" correctly takes less artistic skill than research. ToolArtist makes searching, reasoning, and drawing all actions of a single policy — a fully agentic image generator. We walk through the conversion trick behind its SFT data, the dual-reward RL scheme (RAD-GRPO), and the 0.79 WISE result, straight from the paper.

ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation

Great brushwork, no idea what it's painting

"Draw a woman working in Whitechapel, London, in 1889." Today's text-to-image (T2I) models will hand you something with confident composition and gorgeous texture. But the cobblestones, the gas lamps, the soot-blackened brick — every detail that period accuracy actually hinges on — will be cheerfully wrong. The paper diagnoses the cause plainly: the required knowledge is present neither in the prompt nor in the model's parameters (§1). Generation that depends on culture, history, geography, or a specific IP is what the paper calls open-world image generation, and it formalizes the task as fetching the missing evidence $\mathcal{Z}$ from the world $\mathcal{W}$ on your own (§2.1).

A human illustrator would go look things up before picking up a pen. So let the AI look things up too — and indeed that work already exists, with benchmarks like WISE and WorldGenBench built to measure exactly this gap (§1). What makes ToolArtist new is not that it searches. It is that searching, drawing, and redrawing are all moves of one and the same model.

The two compromises that came before

Existing agentic image generation splits into two shapes, according to the paper (§1, Figure 2).

1. Fixed pipelines: prompt understanding → retrieval → evidence aggregation → image synthesis, with the sequence hard-wired in advance. Order and roles are fixed; the model cannot alter the flow.
2. Prompt optimizers: a search agent gathers evidence, rewrites the prompt, and hands it to a separate image generator. The agent's job ends just before generation begins.

Both help, but they share a weakness: the act of generating the image is never an action the model chose. Noticing a knowledge gap → picking and using a tool → producing the final picture — that entire decision process cannot be learned as a single policy. That is the paper's starting complaint (§1).

The idea: demote drawing to just another action

ToolArtist is built on Emu3.5, a unified multimodal model (UMM) that treats text and images as one and the same token stream (§2.2). Because image tokens come out of the same next-token machinery that continues a sentence, "think, call a tool, draw" can all be interleaved in a single autoregressive sequence.

On that foundation the agent is defined as a ReAct-style loop (§2.3). At each round $t$, the policy looks at the history so far, $\mathcal{H}_{t-1}$, writes a reasoning span $r_t$, and picks an action $a_t$.

The same thing in words: read the whole record of what has happened so far, think one thought, then commit to exactly one move — and the menu of moves has only two entries.

Here $\pi_\theta$ is the model (the policy) and $\mathcal{H}_{t-1}$ is the full record so far — instructions, reasoning, search results, generated images. There are only two families of action. Call a tool (fire a query at TextSearch or ImageSearch and append the returned observation to the history), or draw (emit a caption $g_t$ summarizing the generative intent, followed by the image token sequence $v_t$ itself). Crucially, drawing does not necessarily end the episode: the generated image also lands in the history, so the model can look at its own picture, decide it is still wrong, search again, and redraw (§2.3). How many searches, in what order, and when to put the brush down — none of it is scripted.

The training target is drawn just as sharply. A mask $M_j$ restricts the loss to the tokens the model itself produced — its reasoning and actions ($\mathbf{T}$) — while user instructions and tool-returned observations ($\mathbf{O}$) stay in context but out of the objective (§2.3). Imitating what the environment said does nothing to improve the policy.