Paper Walkthrough: Macaron-V1 — A Frozen Base plus a Mixture of LoRAs, Built to Keep Learning After Launch

A guided tour of Mind Lab's open agent-model family Macaron-V1, straight from the paper: the Mixture-of-LoRA architecture that stacks four specialist adapters on a frozen 744B base and picks one per turn, and the recursive self-improvement loop that trains the model and its harness as a versioned pair.

Macaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRA

The model that stops aging the day it ships

One structural weakness of large language models is that the world moves on the moment they ship. A map app updates when a road changes; a model's weights stay frozen at the day training ended. Retraining a 744B-class model from scratch every time is prohibitively heavy, and cramming chat, coding, and tool use into one parameter space invites another problem the paper calls out: heterogeneous objectives competing through shared parameters can interfere with each other (§1).

Macaron-V1, an open agent-model family released by Mind Lab, is an attempt to break this "ship it and freeze it" pattern. The paper frames the goal as experiential intelligence — the ability to learn from experience accumulated in a real environment and to keep learning after deployment (§1).

Two bets: adaptation and collaboration

The paper organizes everything around two system goals (§1).

The first is adaptation. Instead of improving a model in isolation, Macaron-V1 versions the pair of model plus harness — the harness being the scaffolding that turns model output into actions: tools, prompts, the runtime (see LLM agents fundamentals). Experience from one versioned configuration is evaluated under an external contract and used to construct its successor, recursively.

The second is collaboration, pursued through the Mixture-of-LoRA (MoL) architecture: freeze a large base model, layer specialist LoRA adapters on top, and select exactly one adapter per user turn.

The family has three members. The flagship Macaron-V1-Venti puts four LoRAs (chat, agent, coding, GenUI) on a frozen 744B GLM-5.2 base, with 748B as the release-facing label. Macaron-V1-Tall ships the same four-adapter design on a Qwen3.6 base at roughly 50B, aimed at local deployment. Macaron-V1-Coding-Venti is the single-specialist exception, with the coding LoRA merged directly into the base (§1, §2.2).

Mixture of LoRA: swappable branches on a frozen trunk

LoRA is the standard trick of training a small low-rank delta instead of touching the giant weight matrices themselves (background in our LoRA paper walkthrough). MoL's design principle fits in one line (§2.1):

Cluster tasks that share skills and thinking patterns into one LoRA, and keep tasks whose skills diverge sharply in separate LoRAs.

Two properties fall out. Because the base is frozen, a new capability is added by training and registering another adapter — later specialization can never overwrite what the base already knows. And because the base is shared, specialists trained by different teams, or personalized for different users, can coexist on the same runtime (§2.1).

The similarly named MoE (Mixture of Experts) has a softmax gate distribute weight across experts for every token. MoL instead makes a hard selection — one adapter per turn. In the figure below, lowering the temperature makes the distribution collapse onto a single spike; MoL is that limiting case adopted by design, at turn granularity.

So far, so architectural. The interesting questions are operational: who decides which adapter takes each turn, what that decision costs in latency, and why switching specialists doesn't wreck the KV cache. That is where the paper earns its keep.

The three-step routing loop, once per turn

MoL trains no separate router model. The chat adapter L0 itself reads the incoming request and decides which specialist should handle it (§2.3). Each user turn runs three stages:

1. Route: L0 emits exactly one canonical label (L0–L3) under a constrained-decoding grammar with a 24-token budget.
2. Answer: the selected specialist generates the response.
3. Summary: the specialist writes a recap of what it just did, capped at 192 tokens, stored server-side only.

This loop is not free: in the paper's timing profile, Venti pays 0.54 s for routing plus 0.97 s for the summary — about 32% of the 4.68 s three-hop total (§2.3). Routing accuracy reaches 99.12% on a 6,448-sample trace, but the trace comes from LoRA training data, so the paper calls it an implementation diagnostic, not an estimate of generalization (§2.3). A three-arm quality comparison (direct single adapter, routed, routed with KV reuse) detected no degradation — with the caveat that five seeds per arm cannot establish equivalence (§2.3).

Own-views: the trick that makes routing cheap

The most load-bearing implementation idea is the own-view (§2.4). The conversation history each specialist sees is rebuilt deterministically from an append-only timeline: its own past turns verbatim, every other specialist's turn collapsed into that turn's 192-token summary.

This is where the speed comes from. Because the reconstruction is deterministic, when the conversation returns to a specialist, the prefix of its prompt is byte-identical to the previous visit. The native prefix cache in vLLM or SGLang hits on it, and only the newly appended tail needs prefilling: per-adapter KV-cache reuse emerges with zero engine modification (§2.5; background in our KV cache explainer). The deployment arithmetic is equally crisp: four merged copies of the base would mean holding 2.976T parameters, while MoL holds one base plus adapters, about 774.8B — a 74% reduction (§2.6). The paper compares weight residency only; it claims no latency or throughput advantage over merged deployments (§2.6).

Recursive self-improvement: the training target is the model–harness pair

The second half of the paper is about the training loop. The agent policy is written as (§3.2.1):

Put in words: what the agent does next is decided by conditioning on everything it has seen up to that moment. The bar separates the action being chosen (left) from the evidence it is chosen on (right), and everything after the semicolon is what stays fixed while training happens.

Here $\pi$ is the policy (how the agent picks actions), $\phi$ the trainable LoRA parameters, $a_t$ the action at step $t$, $o_{\le t}$ the observations so far, $\theta$ the frozen base weights, and $c$ the versioned harness configuration. The point of the notation: there are two distinct paths to improvement — update the weights $\phi$, or swap the configuration $c$.

The recursive self-improvement (RSI) cycle runs in three stages (§3.2.3). Discovery: the current model proposes tasks slightly beyond its own competence, each with a verifiable answer or grading rubric. Expansion: those tasks run in the production harness; audits attribute each failure to the model, the task, or the configuration, and candidate edits to prompts or tool exposure are validated by re-running the affected slice. Update: selected trajectories train the LoRA specialists with GRPO, and the accepted configuration becomes the next generation's runtime.

The experiment isolating Expansion alone is the most striking (§3.2.5). Take 122 tasks (from TerminalBench 2.1 source families) that the frozen base fails outright under the official reward. With no weight update at all — only configuration search — cumulative coverage reaches 122/122 after 69 jobs and 450 attempts. The better of two single-configuration full-set sweeps manages only 11/122 (9.0%). The lesson: "the model fails the benchmark" often means "the behavior wasn't elicited," not "the capability is absent." The boundary is stated just as plainly: the search adaptively targeted still-uncovered tasks, so 122/122 is a coverage ceiling for adaptive search, not a generalization estimate for any single configuration (§3.2.5).

Infrastructure: MinT and LongStraw

Two systems keep this loop turning (§4). MinT manages LoRA training over a resident, frozen base and hands off only the exported adapter revision to serving; against a merged-checkpoint handoff, the companion measurement shows up to an 18.3× reduction in the handoff step (§4.1). LongStraw is a response-only long-context RL execution path — gradients flow through response tokens while the prompt is captured without autograd — with completed updates at prompts around 2.1M tokens. The paper flags these as execution receipts, not evidence that Macaron-V1 was trained at those lengths (§4.2).

Results: strong scores, carefully fenced

On the in-house Personal Intelligence suites, Venti scores 58.3 on Macaron ChatBench (GPT-5.5: 55.5) and 64.0 on Macaron LivingBench (Opus 4.8: 63.8). On UI4A-Bench, the generative-UI benchmark, it posts 87.8 against 75.9 for Opus 4.8 and 72.1 for GPT-5.5 — the clearest lead in the table. Its 87.6 on TerminalBench 2.1 is the largest listed value, while on SWE-Verified (85.6) Opus 4.8 stays ahead at 88.6. The small Macaron-V1-Tall beats its Qwen3.6 base on all seven rows evaluated for both (§6.2).

The paper fences its own numbers. ChatBench and LivingBench are built from the company's own product failures and share domains with the RSI loop — home turf, in other words (§5.1). The ChatBench judge is a GLM-5.2-family model, which may favor GLM-derived responses, and the 0.2-point LivingBench margin "should not be interpreted as established superiority" — the paper's own phrasing (§6.2).

Limitations, in the paper's own words

The limitations section is refreshingly direct (§7.2). What shipped is one snapshot of an RSI loop; compounding improvement across generations is not yet demonstrated. The REPL agent harness is not universally best: on stateful observe-before-commit APIs it loses to plain function calling on BFCL v4 (49.5% vs 54.0%, from the companion study), and character-stability degradation appears in very long simulated sessions (§6.3). The report also does not document the consent basis or de-identification audit for the product conversations it uses, nor an independent safety evaluation — and the headline promise of collaboration, composing adapters trained by different teams, remains untested: this release only exercises the four in-house specialists (§7.2).

How this shows up on the job

The readers this paper pays off for most are engineers running LLM serving and ML engineers building internal specialist models.

• For multi-tenant or multi-purpose serving, vLLM/SGLang multi-LoRA support (--enable-lora, --max-loras, --max-lora-rank in vLLM) lets you run "one base, one adapter per use case" today. The own-view design carries a rule worth stealing: keep your prompt prefixes deterministic. If your history-reconstruction logic wobbles between requests, the prefix cache misses everything and inference cost jumps.
• On the fine-tuning side, the knobs are r (rank), lora_alpha, and target_modules. For reference, Venti's public adapters use rank 16 / alpha 32; Tall uses rank 64 / alpha 128 (§2.2).
• Three pitfalls. Switching adapters invalidates the adapter-specific portion of the KV cache, so frequent switching kills cache efficiency (§2.5). A routing hop is a latency tax — about 32% of the total in this paper's configuration (§2.3). And treat any divergence between your training harness and your production harness as a bug to fix, not a model problem to paper over (§3) — the classic reason an agent passes evals and falls apart in production.

Takeaway

Macaron-V1's proposal, in one sentence: move from "build one strong model" to "keep the base, the specialists, and the harness on separate release clocks." The parts are established technology; what's worth studying is the integration — routing delegated to L0's own reasoning, KV reuse made to emerge from deterministic own-views, the harness promoted to a first-class training target. The two headline bets, compounding continual learning and collective intelligence, remain open, as the paper itself concedes. Whether the next release shows cross-generation gains is where this bet gets settled.