Paper Walkthrough: MaP-WAM — Turning a Robot's Memory into Plans So the Executor Stays the Same Size as History Grows

For robot tasks you can't solve without remembering the past, MaP-WAM stops feeding the executor the full video history and instead compresses it into a plan for what to do next. The executor predicts its own progress alongside actions to decide when a step is done. We walk through the design, which reports 83.3% on RMBench and 78.0% on real robots, straight from the paper.

Memory as Plans: World-Action Modeling with Memory-Grounded Planning

A robot that forgets what it just saw

Ever stood over a pot wondering whether you already added the salt? Looking into the pot won't tell you. The answer lives only in what you did a minute ago. Robots hit the same wall.

Most robot policies today (models that map camera images and an instruction to arm motions) assume the current image is enough to choose the next action. That's the Markov assumption. But take a task like "note the color of each button, then find the right one after covers hide them all." By decision time, the information you need is no longer on screen.

The paper is "Memory as Plans: World-Action Modeling with Memory-Grounded Planning" (arXiv:2609.11561, published September 10, 2026, by Sizhe Zhao et al., Harbin Institute of Technology and others).

Here's the abstract in short. Many real manipulation tasks are non-Markovian and need long-horizon memory beyond the current observation. Existing memory mechanisms use language summaries, growing visual windows, or a mix of both. So they either lose fine-grained visual evidence or have to trade history coverage against execution speed. The proposed method, MaP-WAM, splits memory-dependent action generation into memory-grounded planning and plan-conditioned execution. The long history is used only as evidence at planning time. Memory is stored as "completed segment records" (an instruction plus a few sparse frames). From these records the planner builds a compact plan: the next segment's language instruction plus visual guidance. The executor, a World-Action-Progress (WAP) model, predicts actions and execution progress together. It calibrates that progress by matching observations against the plan, which lets it decide on its own when to move to the next segment. The executor's context length is fixed, and KV caching works in both planning and execution. The paper reports a state-of-the-art 83.3% success rate on RMBench, 78.0% on real-robot tasks, and roughly constant executor latency as the task history grows.

An analogy: a travel itinerary vs. a video of the whole trip

Existing approaches look like this:

• Language-diary memory: you jot down "arrived at station," "had lunch." It's light, but details like which platform get lost.
• Record-everything memory: you carry the full video of the trip and rewatch it before every step. The details survive, but rewatching takes longer the longer you travel, and eventually you run out of memory.

MaP-WAM takes a third route. You rewatch the past only when deciding where to go next. While walking, you look at a single itinerary page. That page has a sentence ("head to the museum next") and a few pictures of what the way there should look like. The walker follows only that page. When they feel the page is about 95% done, they ask for the next one. That is what the paper means by memory as plans.

Mechanism 1: split the problem in two

The paper first writes a general memory-dependent policy as (§3.1):

In plain words: from all images so far $\mathbf{f}_{\le t}$, the current joint state $\mathbf{s}_t$, and the instruction $l$, output the next chunk of $h$ actions $\mathbf{a}_{t+1:t+h}$. The trouble is that $\mathbf{f}_{\le t}$ grows without limit.

MaP-WAM splits the work between two parts. A planner produces a plan $\mathbf{C}_k$ for segment $k$. An executor $\pi_{\mathcal{E}}(\mathbf{a}_{t+1:t+h}\mid \mathbf{C}_k, \mathbf{f}_t, \mathbf{s}_t, l)$ sees only that plan plus the current image and state. The paper stresses that the executor's inputs don't depend on history length (§3.1). The history can get as long as it likes, and the executor still reads the same amount.

Use the widget below to feel the gap between a context that keeps growing and one that stays constant (O(1)). MaP-WAM's executor aims for the flat line at the bottom.

Mechanism 2: how memory is stored, and the language-plus-visual plan

The unit of memory is a completed segment record (§3.2). For each segment $i$, the system stores the instruction $l_i$ together with a handful of frames $\mathbf{G}_i$ sampled evenly from the actual execution video. Appendix B gives the details. The segment is split into 8 equal time bins and the last frame of each bin is kept ($N=8$). The very first observation is stored separately as $\mathbf{G}_0$.

Planning happens in two stages (§3.2):