Paper Explainer: Why Agent Skills Work — and Where They Break
Handing an agent a written procedure improves its scores. This paper dissects why, using 8,135 trial records and 528 matched trajectory triples. Skills work as procedural anchors rather than knowledge injection — and they break once the skill library grows.
Demystifying Agent Skills: Why They Work-Until They Don't
Primary source — what this article is built on
undefined2026-08-14→undefined2026-08-22same month
Demystifying Agent Skills: Why They Work—Until They Don'tZhiyuan Jiang, Fangrui Huang, Hanwen Xing et al. · 2026-08-14 · v1arXiv:2608.14036Paper page·PDFundefined
Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge. However, existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: \emph{\textbf{When do skills help, why do they work, and where do they fail?}} Through controlled experiments across various benchmarks, agent harnesses and LLMs, we isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework robustness of skills. To further answer this question, we design a contrastive study that combines controlled quantitative experiments with paired trajectory analysis. We normalize 8,135 trial records from controlled experiments and retain 238 valid unique labels from 240 open-coded records. We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes: skills work when noisy trajectories become procedural anchors that stabilize execution. Skills improve over Workflow Memory by 6.06 points in matched comparisons. Procedural anchoring accounts for 65.7\% of skill cases, versus 4.5\% for explicit knowledge injection, showing that skills stabilize action rather than inject missing facts. Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6\% to 3.3\%. Confusable distractors impair offline identification, yet downstream success remains stable; exact ground-truth invocation is neither sufficient nor necessary. Skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation. These findings move evaluation beyond aggregate success rates and guide reliable self-evolving agents.
A skill is a recipe card
Picture asking someone to cook a dish. You can hand them nothing and let them improvise. You can hand them the full transcript of a previous attempt — including the batch that got over-salted and thrown out, and the three trips to the fridge. Or you can hand them a single recipe card distilled from that transcript.
What the LLM agent world calls a skill is the third option. In the paper's framing, a skill is not a record of past execution but a compact description of what to do, what to check, and which pitfalls to avoid (§1). The second option corresponds to Workflow Memory: past trajectories, merely cleaned up and passed along.
That contrast is the crux. Both artifacts are built from exactly the same experience; only the packaging differs. So comparing them separates "the agent got better because it saw prior experience" from "the agent got better because of how that experience was written down." Maybe the recipe card wins not because it contains more, but because it contains less — that is what the paper sets out to test.
Measuring "why it works," not "whether it works"
Existing work on skills has looked almost exclusively at aggregate success rates: if a skill-augmented agent solves more tasks, the skill is useful. The paper argues this is not enough. An aggregate number says nothing about what actually changed in the agent's behavior once a skill was loaded, or why the same skill helps one task and hurts another (§1).
The alternative is a controlled comparison: run the same task under three arms — Raw (no prior experience), Workflow Memory, and Skill — and diff the resulting trajectories. The authors normalized 8,135 trial records into a shared manifest (7,837 of them retain an agent transcript), open-coded 240 sampled trajectories, and consolidated 238 valid unique labels into three high-level categories and twelve modes (§4).
The taxonomy is not left to the model's word. A human annotator inspected three supporting trajectories for each of the 238 labels (714 checks in total) and then independently re-mapped all labels to the twelve modes: 95.8% exact agreement, Cohen's κ = 0.952 (§4, Table 3).
The experimental skeleton: one experience, two wrappings
For each task the authors collected both successful and failed trajectories and built a fixed-budget composition grid — six mixtures from 5s0f (five successes, zero failures) to 0s5f. Workflow Memory and Skill are constructed from the same mixture and evaluated on the same tasks, so only the representation varies (§3.3). Benchmarks are Terminal-Bench 2.0, Terminal-Bench-Pro and SkillsBench; the agent–model pairings are Codex + GPT-5.3-Codex and Gemini CLI + Gemini-3.1-Pro-Preview (§3.2).
One row from Table 1 sets the tone. On Terminal-Bench 2.0 with Codex, Raw scores 0.5935 and Skill at 5s0f reaches 0.7548 — while Workflow Memory built from that identical material sits at 0.4452, below the baseline. Even at 0s5f, where every source trajectory is a failure, Skill leads 0.5161 to 0.2839.
The five gates in a skill's life
The picture the paper ultimately draws treats skills as a lifecycle rather than a memory-injection trick (§5.3). To pay off, prior experience must be ① distilled at the right granularity, ② stored, ③ retrieved for the situation at hand, ④ invoked by the agent, and ⑤ adapted to the current context. Fail any one gate and the right skill sitting on the shelf yields nothing.
Gate ③ works by embedding the task instruction and each skill description as vectors and ranking by how closely they point in the same direction (the paper uses Qwen3-Embedding-0.6B, §3.2). The yardstick is the dot product and cosine similarity — the figure below gives you a feel for it. Note how two vectors can score highly while describing quite different things; that property matters later.
Comments
Sign in to comment