Paper Walkthrough: EnvACE — Agents That Rehearse the World Instead of Calling It

EnvACE trains a tool-using LLM agent without ever calling an external environment: the same policy alternates between acting and playing the environment. We walk through role-wise GRPO, test-time private rehearsal, results on four agent benchmarks, and the limitations.

EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning

Can you get stronger by shadow practice alone?

A chess master can play out an entire game in their head, no board required. They can do this because the likely replies to each move are already inside them: a strong player has internalized not just how to choose moves, but how the position responds to their moves.

EnvACE, the paper we're reading today, brings exactly this idea to training LLM agents — AIs that accomplish tasks by calling external tools. Conventional agent training has the model learn by interacting, over and over, with a live API or database. EnvACE instead makes the model itself play the environment's part, a mechanism the authors call world rehearsal, so that training never touches an external environment at all. The headline claim: across four agentic benchmarks, this beats prior methods that invest heavily in building environments, in the overall evaluation (Abstract).

The problem: sparring partners are expensive

To train a tool-using agent with reinforcement learning (RL), you need an environment response after every action. The agent emits a tool call, gets a result back, decides its next move; the loop produces a trajectory, and a task-success reward arrives at the end. The question is: who produces those responses?

The paper sorts existing answers into two camps (§1).

1. Build executable environments — real or synthesized APIs and databases. Responses are accurate, but construction and verification are costly, and verifying correctness gets harder as environments grow more complex.
2. Let an LLM simulator answer — cheap and flexible, but responses can be inaccurate or inconsistent, and grounding them still ends up requiring real environments.

Either way, training stays dependent on an external response provider, and the ability to model the environment lives outside the acting policy. The authors argue that a good agent policy should not only act to solve the task but also model how the environment reacts to its actions (§1).

The twist: one model, two roles

EnvACE's answer is simple: assign two roles to a single policy (the same LLM) (§4.1).

• Act: read the interaction history and generate a tool call
• Rehearse: given the tool call it just produced, generate the response the environment would return

The rehearsed response is appended to the history, and the next action is chosen on the basis of that self-made response. Act, respond, act, respond — the whole trajectory is spun out by the model alone. It is no longer a dialogue with the outside world; it is, in the paper's words, a process unfolded by the policy itself. The roles are switched by role prompts, but the parameters are shared. As training repeats, the mapping "if I do this, the environment answers like that" gets written into the weights, and the policy itself becomes an agent world model — which is what the title's "internalizing" means.

Both roles are just LLM token generation under the hood, so how varied the outputs are is governed by sampling temperature. The paper indeed uses different temperatures per role and per experiment — 1.0 for both roles where they want broad exploration of the action space, 0.01 where reproducibility matters (§5.1).

Following the mechanics in equations

The paper formalizes a tool-interactive task as a finite-horizon POMDP (partially observable Markov decision process) (§3). At each step $t$, the policy $\pi_\theta$ picks an action from the history $h_t$, and normally the environment would return an observation. EnvACE hands that job to the policy as well (§4.1). First, the acting role moves.