Paper Walkthrough: EnvACE — Agents That Rehearse the World Instead of Calling It

EnvACE trains a tool-using LLM agent without ever calling an external environment: the same policy alternates between acting and playing the environment. We walk through role-wise GRPO, test-time private rehearsal, results on four agent benchmarks, and the limitations.

EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning

Can you get stronger by shadow practice alone?

A chess master can play out an entire game in their head, no board required. They can do this because the likely replies to each move are already inside them: a strong player has internalized not just how to choose moves, but how the position responds to their moves.

EnvACE, the paper we're reading today, brings exactly this idea to training LLM agents — AIs that accomplish tasks by calling external tools. Conventional agent training has the model learn by interacting, over and over, with a live API or database. EnvACE instead makes the model itself play the environment's part, a mechanism the authors call world rehearsal, so that training never touches an external environment at all. The headline claim: across four agentic benchmarks, this beats prior methods that invest heavily in building environments, in the overall evaluation (Abstract).

The problem: sparring partners are expensive

To train a tool-using agent with reinforcement learning (RL), you need an environment response after every action. The agent emits a tool call, gets a result back, decides its next move; the loop produces a trajectory, and a task-success reward arrives at the end. The question is: who produces those responses?

The paper sorts existing answers into two camps (§1).

1. Build executable environments — real or synthesized APIs and databases. Responses are accurate, but construction and verification are costly, and verifying correctness gets harder as environments grow more complex.
2. Let an LLM simulator answer — cheap and flexible, but responses can be inaccurate or inconsistent, and grounding them still ends up requiring real environments.

Either way, training stays dependent on an external response provider, and the ability to model the environment lives outside the acting policy. The authors argue that a good agent policy should not only act to solve the task but also model how the environment reacts to its actions (§1).

The twist: one model, two roles

EnvACE's answer is simple: assign two roles to a single policy (the same LLM) (§4.1).

• Act: read the interaction history and generate a tool call
• Rehearse: given the tool call it just produced, generate the response the environment would return

The rehearsed response is appended to the history, and the next action is chosen on the basis of that self-made response. Act, respond, act, respond — the whole trajectory is spun out by the model alone. It is no longer a dialogue with the outside world; it is, in the paper's words, a process unfolded by the policy itself. The roles are switched by role prompts, but the parameters are shared. As training repeats, the mapping "if I do this, the environment answers like that" gets written into the weights, and the policy itself becomes an agent world model — which is what the title's "internalizing" means.

Both roles are just LLM token generation under the hood, so how varied the outputs are is governed by sampling temperature. The paper indeed uses different temperatures per role and per experiment — 1.0 for both roles where they want broad exploration of the action space, 0.01 where reproducibility matters (§5.1).

Following the mechanics in equations

The paper formalizes a tool-interactive task as a finite-horizon POMDP (partially observable Markov decision process) (§3). At each step $t$, the policy $\pi_\theta$ picks an action from the history $h_t$, and normally the environment would return an observation. EnvACE hands that job to the policy as well (§4.1). First, the acting role moves.

Here $a_t$ is the action at step $t$ (a tool call or a final answer), $h_t$ is the interaction history so far, and $\textsf{Act}$ is the role instruction "behave as the actor." Read in words, the line says: look at everything that has happened so far, put on the actor's hat, and produce the next move. The $\sim$ is worth noticing too — the move is sampled rather than fixed, so the same history can yield different actions on different rollouts. Then the rehearsal role writes the reply.

$\hat{o}_t$ is the predicted environment response — the hat marks it as home-made, not a real observation $o_t$. The same model, now wearing the environment's hat, writes what this tool call would get back — a line which says, in everyday terms: given the history and the move I just made, guess the reply I would have received. Nothing leaves the machine; the answer is written by the very model that asked the question. Finally the history is updated.

$\oplus$ means appending to the history. Put in words: the history for the next step is the old history with my move and my own guess at the reply stapled onto the end. The action and its rehearsed response are added as a pair, the next action is decided by reading them, and this act–rehearse alternation continues until termination — one full trajectory, zero calls to any external environment.

Learning: GRPO with per-role baselines

Once trajectories exist, how do you learn from them? EnvACE samples $K$ rollouts for the same instruction $x$ and assigns each trajectory a single task-outcome reward $R_i$ (from a verifiable evaluator or a checklist-based LLM judge, §4.2). Every output inside a trajectory inherits that reward — but the key design choice is that the advantage (how much better an output was than average) is measured against a role-specific baseline.

$\mathcal{G}_{x,r}$ collects, across the $K$ rollouts, all outputs generated under role $r$ (Act or Rehearse); $\mu_{x,r}$ is the mean reward for that role; and $A_{i,m}$ is the advantage of output $y_{i,m}$. The left half says, in words: average the rewards of everything this role produced across the $K$ rollouts; the right half then says: an output's score is its own trajectory's reward minus that role's average. Concretely: actor outputs are compared with other actor outputs, environment outputs with other environment outputs. The baselines are separate because the two roles affect the reward differently — but both roles update the same shared parameters $\theta$, which is precisely how knowledge gained while rehearsing flows into the acting side. Optimization uses the standard clipped GRPO objective (Eq. 8, §4.2).

At test time: rehearse privately, then commit once

A policy that has internalized a world model has a second use at inference (§4.3). Given a new task, it first runs $N$ private rehearsal attempts without touching the external environment — imagining an act–rehearse trajectory, then evaluating it to produce an assessment and a revision suggestion. Two modes:

• Parallel: the $N$ attempts are imagined independently
• Sequential: each new attempt sees the earlier imagined trajectories and their critiques, so it can avoid repeating failures

All rehearsals are then summarized into a compact rehearsal memory $m_x$, and the agent performs a single committed execution in the real environment, conditioning on that memory. The rehearsals stay private and never alter the external environment's state.

How well does it work?

The main experiments train Qwen3-8B on the CM2 dataset for 470 steps (using the verl framework on 16 H20 GPUs, §5.1), evaluating on BFCL-v4, $\tau^2$-Bench, VitaBench, and FinMCP-Bench.

• Overall: 32.91% averaged over the three main benchmarks, above every environment-scaling baseline with complete results (+0.99 over EnvScaler-8B, +0.37 over AWM-14B, §5.2). That breaks down as 36.7% average on $\tau^2$-Bench, 16.0% on VitaBench (best in the 7–8B class), and the best TF1 of 46.78% on FinMCP-Bench.
• Does rehearsal itself help? Against standard GRPO under controlled conditions, the $\tau^2$-Bench average rises from 31.2% to 36.7% (+5.5) (§5.3).
• Does parameter sharing matter? A variant with separate models for acting and rehearsal (Per-role Policy) reaches 35.5%, versus 36.7% for the shared version (+1.2) — supporting the hypothesis that internalization happens because both roles write into the same weights (§5.3).
• Model scale: applying the same recipe to 1.7B and 8B backbones, the gains grow with scale ($\tau^2$ average 15.3% → 36.7% going from 1.7B to 8B, §5.4).
• Test-time scaling: with a rehearsal budget of $N=2$ in parallel mode, Overall improves from 36.7% to 40.9% (+4.2). Crucially, letting the untrained base model do the rehearsing barely helps — and in sequential mode it falls below the no-rehearsal baseline (34.9%). The paper reads this as evidence that the gains come from the internalized environment knowledge, not from extra inference compute per se (§5.5).

The case studies in Appendix A are telling. EnvACE predicts during rehearsal that a planned tool call will fail due to invalid parameters and repairs them before execution, and in another case detects that a write operation is not allowed and swaps it for a read-only query — while baseline agents actually fail first and then spend extra steps recovering.

Limitations, and how to read the claims

The authors name two limitations (§7): compute constraints mean EnvACE is verified only up to the 8B scale, and the evaluation focuses on tool-interactive tasks. Within the paper's own results, note also that test-time rehearsal degrades at $N=3$ compared to $N=2$ — the imagined trajectories lengthen the input toward the effective context limit, the authors suggest (§5.5). More rehearsal is not automatically better.

One caveat from outside the paper (our practical reading): a rehearsed response is the model's imagination, with no correctness guarantee from any real environment. It cannot be expected to capture quirks of environments unlike anything in its training data (CM2-derived tasks, in EnvACE's case).

How this is used in practice

The people this paper speaks to most directly are ML engineers building RL training infrastructure for agents. Environment-synthesis pipelines (the EnvScaler / ScaleEnv line of work) require per-environment construction and maintenance; world rehearsal removes the environment call from the rollout loop entirely, so the training stack shrinks toward "a model and a reward function." Implementation-wise, you work in an RL framework like verl (as the paper does), and the core change is in GRPO advantage computation: tag every output with its role and normalize against per-role baselines. The knobs you actually touch are rollouts per prompt (4 in the paper), the KL coefficient ($10^{-4}$), per-role sampling temperatures, and — on the inference side — the rehearsal budget $N$ and its mode (parallel vs. sequential).

If you tune inference for an agent product instead, test-time rehearsal maps directly onto "dry-run before hitting the real API." The Appendix A cases show where it pays off most: rehearse once before any write-type tool — reservation changes, payments, anything irreversible.

Three pitfalls. First, you need hard separation so that rehearsed tool calls can never reach real APIs — keep the paper's structure of private rehearsal plus exactly one committed execution. Second, don't crank $N$: the paper sees a drop at $N=3$, so pick the budget with your context length in mind. Third, the rehearsing model must be one trained this way; letting a vanilla model imagine trajectories gives little benefit and can hurt in sequential mode. "Add test-time scaling and everything improves" is not what the evidence shows.

Read next

• LLM Agents from First Principles — the tool-call-and-observation loop this whole paper builds on
• Instruction Tuning and RLHF — the foundations of updating an LLM from rewards
• RAG Fundamentals — contrast how retrieval pipelines vs. agents handle the outside world