Paper Explained: EVISKILL — Rewriting an Agent's Playbook from Replayable Evidence
A framework for automatically revising an LLM agent's skill document that pins every proposed edit to the exact trajectory steps that motivated it, then re-runs only those steps to check the edit actually works. It posts the best accuracy in 14 of 18 benchmark–model settings.
Key points
- EVISKILL, from a team at Jilin University, is a skill-evolution method that rewrites a skill, a Markdown playbook, automatically from execution logs.
- Existing methods tend to lose the behavioral evidence behind each edit and judge a revision only as a whole, so a locally correct fix can be thrown away with a rejected revision.
- EVISKILL organizes execution observations into Replayable Evidence Cards, synthesizes edits that keep explicit links to that evidence, and verifies them by re-executing the targeted steps.
- Supported edits are kept provisionally across epochs for further refinement, while global validation still decides what enters the final skill.
EVISKILL: Grounding Skill Evolution in Replayable Evidence
Primary source — what this article is built on
undefined2026-10-04→undefined2026-10-10same month
EVISKILL: Grounding Skill Evolution in Replayable EvidenceYan Zhou, Yili Wang, Yiwei Dai et al. · 2026-10-04 · v1arXiv:2610.05030Paper page·PDFundefined
Continual skill evolution enables LLM agents to accumulate and refine reusable procedural knowledge from interaction experience without updating model parameters. Its effectiveness depends on determining not only what to change, but also why a change is justified and when it should become persistent guidance. However, existing experience-driven methods can lose the behavioral evidence and task contexts supporting edits. Moreover, a global validation outcome provides an incomplete judgment of its constituent changes: locally supported corrections may be discarded with a rejected revision, while evidence may require further experience to inform useful updates. To this end, we introduce EVISKILL, an evidence-driven framework that organizes execution observations into Replayable Evidence Cards and synthesizes edits with explicit links to their supporting contexts. Targeted replay verifies these edits through re-execution and provides feedback for correction. Across epochs, EVISKILL preserves evidence and provisionally retains supported edits for further refinement, while global validation governs their incorporation into the final skill. Experiments on three interactive benchmarks across six LLM backbones demonstrate the effectiveness of this approach.
Who edits the agent's playbook?
Modern LLM agents often load a skill: a Markdown playbook, separate from the model, that says which tools to call, in what order, what to check first, and how to recover. Because a skill changes behavior without touching weights, a growing line of work rewrites skills automatically from execution logs (§1). This article covers "EVISKILL: Grounding Skill Evolution in Replayable Evidence" (arXiv:2610.05030, posted October 4, 2026) by a team at Jilin University.
Here is the abstract in plain terms. Evolving a skill means deciding not only what to change but why a change is justified and when it should become permanent guidance. Existing experience-driven methods tend to lose the behavioral evidence behind each edit, and they judge a revision only as a whole, so a locally correct fix can be thrown away along with a rejected revision. EVISKILL organizes execution observations into Replayable Evidence Cards, synthesizes edits that keep explicit links to that evidence, and verifies them by re-executing the targeted steps. Supported edits are kept provisionally across epochs for further refinement, while global validation still decides what enters the final skill.
An analogy: the manual-revision meeting
Picture a factory revising its operating manual. The usual routine: collect suggestions like "we had this failure last week," bundle them into a new edition, trial it, and if the numbers drop, scrap the whole edition.
That routine has two holes. Nobody recorded which job and which step "last week's failure" was, so nobody can check whether the new wording actually prevents it. And if one of ten changes drags results down, the other nine get scrapped with it.
EVISKILL sticks a note on every suggestion saying "log X, steps s through t," then re-runs just that scene with the new wording. Even when the edition is rejected, suggestions that worked in their re-run go onto a holding shelf and come back for review at the next meeting.
Why "learn from experience" is not enough
The authors start with two preliminary studies (§2). First, they took edits that the prior method Trace2Skill derived from trajectories, applied each one alone, and re-ran the tasks that motivated it. Outcomes were mixed: some edits helped, some changed nothing, some made things worse (§2.1). Second, picking only the promising edits out of revisions that failed validation improved results over the previous skill (§2.2). An edit is not effective just because it came from experience, nor useless just because its revision was rejected.
The paper also warns that a fix induced from one failure may overfit the observed case (§1). Push the degree up in the widget below: training error keeps falling while test error pulls away. A skill edit tailored to the failure in front of you is prone to the same trap on tasks you haven't seen.
The paper closes those holes in three stages: (1) build edits tied to evidence, (2) verify each edit by replaying the scenes that motivated it, and (3) carry working edits and unresolved evidence forward even out of rejected revisions (§1). Let's take them in order.
Comments
Sign in to comment