Paper Walkthrough: Miles v0.1 — Anatomy of a Post-Training Stack Built to Survive Production

A read-through of Miles, a full-stack system for agentic RL post-training of frontier MoE models: how it hunts down the numerical drift between the rollout engine and the trainer, and what a 744B model actually measured on 64 GPUs.

Miles v0.1: Production-Level Post-Training

What the paper claims

The original title is "Miles v0.1: Production-Level Post-Training" (RadixArk, arXiv:2609.08368, published 8 September 2026).

Here is the abstract in plain terms. Miles is a full-stack, production-ready system for frontier post-training — the stage that turns a pretrained model into a useful one. Building on the clean design of its predecessor slime, it reorganizes every stage of the reinforcement-learning (RL) loop around one principle: components should be verified, clean, and customizable. With accuracy, efficiency, reliability, and scalability as first-class goals, it ships rollout engines built on SGLang, a trainer with a choice of two backends (NVIDIA Megatron-LM and PyTorch FSDP), and three weight-synchronization transports for different deployment topologies. Beyond full-parameter RL it supports LoRA RL, on-policy distillation, supervised fine-tuning, and true-on-policy rollout–training alignment, and carries the same architecture over to diffusion models. It closes with an end-to-end case study: fully asynchronous agentic RL on GLM-5.2 (744B-A40B) over terminal-use coding tasks, on 64 NVIDIA GB300 GPUs, with a median step time of 263 seconds over the first 30 measured steps.

So this is not an algorithms paper — it is an infrastructure report. Its value lies in listing, from hands-on experience, what actually breaks at frontier-scale RL.

An analogy: two orchestras reading the same score

RL post-training is, crudely, "let the model write answers, score them, push the score back into the weights." But the side that writes the answers (the inference engine) and the side that fixes the weights (the trainer) are different pieces of software.

Hand two orchestras the same score — the same weights — and they will not produce quite the same sound. Different kernels, different precision, different batch shapes. Both performances are legitimate, yet the number the scorer estimated for "how likely was this player to produce that note" disagrees with the probability under which the note was actually produced. The paper calls this train-rollout mismatch, and the whole system is designed around killing it (§1). What makes it nasty is that it raises no error and merely rots the gradient (§2).

Three stages — and why running them in lockstep wastes hardware

The Miles loop has three stages (§1.1).

1. Rollout: SGLang engines generate trajectories. In agentic RL a trajectory is an entire multi-turn episode, and each session talks to its own isolated environment (a sandbox) that produces the reward.
2. Training: Megatron-LM or FSDP consumes completed trajectory groups, computes the RL loss, and updates the policy.
3. Weight update: the new weights are synchronized to the rollout engines.

One term is worth pinning down. A group is the set of trajectories generated from the same prompt. Objectives like GRPO score a trajectory by comparing it against the other attempts at the same prompt, so trajectories sharing a prompt must be generated and consumed together. Miles therefore schedules, buffers, and discards data one group at a time rather than one trajectory at a time (§2).

Run those three stages in turn and hardware idles on both sides. The trainer waits for the slowest trajectory in the batch; then the engines wait for the optimizer. On long-context and tool-using tasks, that wait dominates the batch. Hence Miles offers a fully asynchronous mode: engines keep generating, and the trainer forms a batch from whatever has finished (§2.2). That requires GPU capacity for both stages at once, so it presumes a disaggregated placement with separate pools. Ask for fully asynchronous mode on shared GPUs and Miles refuses to start.

Going asynchronous manufactures stale data

Decouple generation from training and the weights a trajectory was produced under drift away from the weights it is trained on. The paper calls that gap staleness, and defines it as "the current trainer weight version minus the oldest weight version appearing anywhere in the group" (§2.2.2). A multi-turn trajectory spans several weight updates while it generates, so tokens inside one group belong to different generations. Miles deliberately anchors to the oldest token — a pessimistic definition.

Between generation and training sits a bounded buffer where finished groups wait. The buffer drops a group for three reasons (§2.2.2, Table 1).

The first two are properties of the group itself, so they can be judged on arrival; staleness depends on how long it waited, so it can only be judged at the moment the trainer collects it. That asymmetry shapes the design.

The paper devotes a section to observability for the same reason (§2.2.3). Under an asynchronous schedule the two stages advance at independent rates, and one can outrun the other without crashing — silently wasting GPUs. Miles reports queue_size, avg_staleness, max_staleness and friends on every training step. A queue pinned at zero means generation cannot keep up; a queue pinned at capacity means the trainer is the constraint, and the staleness of the waiting groups climbs alongside it.

Killing the mismatch, part 1: make the tokens identical (TITO)

Here is the heart of the paper. Miles separates structural causes of the mismatch from numerical ones and attacks them in three stages. Start with the tokens themselves changing (§2.4). In a conventional multi-turn pipeline the model's output passes through message parsing, tool execution, and chat-template rendering before the next turn begins. Along the way tokenization can shift, reasoning content can be pruned, tool calls can be reserialized. The result: the trainer takes an update with respect to a trajectory that never occurred. Miles moves ownership of tokenization from the harness to the session server, checkpoints the exact token IDs, log-probabilities, and routed experts at each completed turn, and tokenizes only the appended suffix on later turns. Token fields an agent supplies itself are overridden.

There are two rules for extending a session (§2.4.1). Linear allows only the tail to grow, so one session yields exactly one training sequence. Branching keeps the history as an append-only tree, and each leaf yields a sequence. A harness that forks or compacts its own context mid-run — Claude Code is the paper's example — does not know in advance how many trajectories a session will produce, and therefore cannot be served by the linear rule. It is a pleasingly concrete observation.

All of this depends on getting the chat template and tool-call parsers right, so Miles never auto-detects them from a checkpoint. The user names a "model family" at launch, and that name resolves to a registration a maintainer has already checked (§2.4.3). Each registration is guarded by two checks: one on CPU, confirming that rendered token sequences stay append-only from turn to turn, and one on GPU against a live model, confirming the same invariant survives real inference. The CPU check alone is not enough — a template can look append-only in isolation and still break the moment a real parser consumes the model's output.