Paper Explained: Repo-To-Skill — Distilling GitHub Repositories Into Skills an AI Can Use

A third layer that is neither the model nor the harness: operational knowledge, distilled from 1,000 GitHub repositories into 5,353 verified skills. What DisCo and the AREX-Skill Library actually do, what 31.11%→72.89% on MLE-bench is made of, and where the skills failed.

Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills

Knowing a method and making it work are different things

Picture an engineer on their first day. They have read the textbooks and can explain the algorithms. They still cannot ship anything, because they do not know which internal library to reach for, what shape of data its API expects, or which setting quietly corrupts a training run. The knowledge that matters is the sentence a colleague drops over your shoulder: "that will fall over as written." It is not in any textbook, and it decides most of the outcome.

The paper here is "Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills" (arXiv:2609.02749, published 2026-09-02, Beijing Academy of Artificial Intelligence with USTC, Renmin University, and Hong Kong PolyU). It is a 48-page, 3-figure technical report; the code lives at github.com/VectorSpaceLab/AREX-Skill.

Here is the abstract in plain terms. Agents that run machine-learning research end to end have been described as two parts: a model backbone, and a harness for planning, execution, memory, and verification. That architecture leaves domain-specific know-how outside the agent. The authors name the missing layer operational knowledge — the know-how that separates knowing a method from making it work. The knowledge is not absent from the field; it lives in repositories and papers, but written for human readers and far too large to load during a task. Distil it once into compact, verified skills, and it can be reused across tasks instead of rediscovered on every run. The authors present DisCo, a research agent that both creates skills and researches with them. Distillation runs in two complementary forms: task-agnostic, condensing widely used repositories into reusable skills, and task-oriented, producing the skills a concrete task calls for. Applying the former across the open ecosystem yields the AREX-Skill Library: 5,000+ verified skills distilled from 1,000 widely used ML repositories, organized into 20 areas and 178 capability families. With the GPT-5.5 backbone, research harness, and downstream execution budget held fixed, the skill-equipped agent scores 134.3% higher on MLE-bench, 34.4% higher on PaperBench, 9.2% higher on FrontierCS, and 14.0% higher on PassNet than the same agent without skills.

A third layer that is neither the model nor the harness

The paper first formalizes a research task as $\tau=(q,\mathcal{D},\mathcal{E},g)$ (§2.1): $q$ is the problem statement, $\mathcal{D}$ the data and material handed over with it, $\mathcal{E}$ the environment including the tools it exposes and the budget it bounds, and $g$ the target the outcome must meet. In plain words: what to solve, with what, where, and how far.

An agent that carries this out on its own has conventionally been written $\mathcal{A}=(M_{\theta},H)$ — the LLM backbone $M_{\theta}$ supplying understanding, reasoning, planning and execution, and the harness $H$ supplying orchestration, memory, verification and iterative refinement. The paper's point is that ML research is expertise-intensive: success depends on knowing which methods and tools to use, when, and how to use them correctly — and neither component carries that expertise. The backbone's prior is broad but fixed; the harness controls procedure but supplies no domain content (§2.2). So a third term is added.

What this says, in words: stop counting an agent as two parts — the brains and the procedure — and count a third, what it already knows about the domain on its first day. One term is added; the other two are left exactly as they were.

$\mathcal{K}$ is the operational knowledge, handed to the agent as explicit operating context. Action selection becomes $a_{t}\sim\pi_{\theta}(a\mid\tau,h_{t},H,\mathcal{K})$: alongside the history $h_t$ and the harness, $\mathcal{K}$ now informs the choice. The control loop is not changed at all. The authors return to this repeatedly: the harness governs how an agent researches, while operational knowledge determines what it knows when research begins (§1, §7).

Without $\mathcal{K}$, the paper identifies two losses (§1). Within a task, the agent must infer package behavior by trial and error, and misconfigurations surface only after budget has been spent. Across tasks, those discoveries are not retained as reusable context. The agent pays the same tuition over and over. In the real library, of course, all 5,353 skills are never loaded at once — a router pulls only the branch nearest the task (§4.2). The figure below gives that "pull only what is close" feeling directly.

The container: a three-layer folder

The paper instantiates $\mathcal{K}$ as skills in the Agent Skills format (§3.1). One skill has three layers.

What this says, in words: a skill is a cover page, a reference shelf, and a toolbox. Only the cover page is always read; the shelf and the toolbox are opened when a task actually calls for them.

SKILL.md is the knowledge interface. It is the only layer read up front, stating what the skill is for, when it applies, and how to proceed — goals, key concepts, tool usage, worked examples, known failure modes. references/ is the knowledge substrate: API documentation, algorithmic detail, parameter configurations, loaded only when needed. scripts/ is the execution interface: executable wrappers with defined inputs and outputs that the agent invokes rather than reimplements.

The split works because the agent reads only the opening and unfolds the rest on demand (progressive disclosure), so it can hold thousands of skills without crowding its context. Since one source usually contains more than one skill's worth of know-how, the skills distilled from a single source are bundled into a skill graph $\mathcal{G}=(\mathcal{S},\mathcal{L})$. An entry skill states the source's scope and links out to component skills for each function or stage; links encode routing, dependency, or composition relations. The agent reads the entry, follows the links its problem calls for, and leaves the rest unopened (§3.1).