Paper breakdownAutonomous AI AgentsFREE2026

Repo-To-Skill

Distilling GitHub Repositories Into AI4AI Skills

Jianlyu Chen, Yuyang Hu, Hongjin Qian, Jiawei Liu, Wenqing Wei, Xiaolong Chen,

Read the Paper on arXiv

Concepts

Formulas

Figures

Demos

Paper Overview

Repo-To-Skill (Chen, Hu, Qian et al. — Beijing Academy of Artificial Intelligence, USTC, Renmin University and Hong Kong Polytechnic, arXiv 2026) starts from an observation about where autonomous ML-research agents actually fail. These agents are built from two parts: a model that reasons and plans, and a harness that orchestrates, remembers, and verifies. Both keep improving — and yet the agent still stumbles on things a competent practitioner would never get wrong, because the domain-specific know-how that separates knowing a method from making it work lives in neither part.

The paper names this missing layer operational knowledge and argues it already exists — scattered across repositories, papers, and blog posts — but in forms written for humans and far too large to load during a task. The proposal is to distill that material into compact, verified Agent Skills: a SKILL.md entry point over reference documents and executable scripts, revealed to the agent only when a task calls for them.

The system that does this is DisCo, a single agent that both creates skills and researches with them. Run across the open ecosystem, its creator mode produces the AREX-Skill Library: 5,353 verified skills distilled from 1,000 widely used ML repositories, organized by a router over 20 areas and 178 capability families.

The evaluation is a clean ablation. Holding the GPT-5.5 backbone, the Codex research harness, and the downstream execution budget fixed, the only variable is whether the agent has the distilled skills. With them, it scores 134.3% higher on MLE-bench, 34.4% higher on PaperBench, 9.2% higher on FrontierCS, and 14.0% higher on PassNet — gains that come purely from adding operating context, not from a stronger model or a new control loop.

Explainer videoThe whole paper, animated

Chapter Roadmap

Click any topic to jump in

Interactive demo

The Missing Layer

A research agent is a model plus a harness — and neither carries the domain know-how that turns knowing a method into making it work.

packaged as

Skills and Skill Graphs

Package operational knowledge as agent-facing skills — a SKILL.md interface over references and scripts — linked into a graph the agent reads on demand.

produced by

Skill Distillation

Rework declarative sources into skills through one four-stage loop: scope, ground, construct, verify. No skill is admitted unverified.

orchestrated in

Creator and Researcher Modes

DisCo both writes skills (creator, paid once per source) and researches with them (researcher, paid per task) over the same backbone and harness.

scaled into

The AREX-Skill Library

5,353 verified skills from 1,000 ML repositories, routed over 20 areas and 178 capability families.

evaluated by

Evidence and caveats

Results

With backbone, harness and budget held fixed, skills lift MLE-bench 134.3%, PaperBench 34.4%, FrontierCS 9.2%, PassNet 14.0%.

When Skills Help

Gains concentrate on hard, low-baseline tasks and are uncorrelated with extra token spend — but two PaperBench tasks regress on retrieval-precision failures.

Concept by concept

7 sections, in reading order.

01 / 07The Missing Layer: Operational Knowledge

Ask an agent to improve a model on an unfamiliar dataset and it must decide which method suits the problem, which package implements it, how that package wants the data laid out, which configurations are appropriate, and which pitfalls silently invalidate an otherwise plausible run. It can infer all of this by trial and error — propose a plan, run it, inspect the failure, revise. The trouble is that those failures spend the same budget reserved for evaluation, and a single misconfigured run can burn a large share of it before producing any meaningful measurement.

The paper formalizes a research task as $\tau = (q, D, E, g)$ — a problem $q$ , its data and material $D$ , an environment $E$ that exposes tools and bounds a budget, and a target $g$ the result must meet — and the standard agentic system as $A = (M_\theta, H)$ : an LLM backbone and a harness. The backbone supplies understanding, reasoning, planning, and execution; the harness supplies orchestration, memory, verification, and iterative refinement. Almost all recent progress has come from strengthening one or the other. But for research tasks, the two-component view leaves something unspecified: the backbone's prior is broad but fixed, and the harness controls procedure but supplies no domain content.

That gap is what the authors call operational knowledge — the expertise that binds the field's methods and tools to the task at hand by specifying what can solve it, when each candidate applies, and how it should be used. They write the research agent as $A_{\text{res}} = (M_\theta, H, K)$ , adding an explicit operating context $K$ . Operational knowledge has two constituents that must travel together: one turns knowledge into capability (methods, code, models, and APIs packaged into things the agent can actually invoke), and the other turns capability into usage policy (the conditions, reasons, and procedures that govern each unit's use). Capability without policy is a tool with no selection criteria; policy without capability is advice with no executable interface.

Crucially, this is not the harness in disguise. The harness specializes how an agent explores; $K$ specializes what it knows to consider. And declarative sources — a paper reporting that a technique improves accuracy, a repo documenting what an API accepts — state what holds, never a course of action for a given problem. The central methodological problem is to derive operational knowledge from that declarative material automatically and at scale.

The Missing Layer: Operational Knowledge diagram

Key Points

1
A research task is formalized as $\tau = (q, D, E, g)$ and the conventional agent as $A = (M_\theta, H)$ — a backbone plus a harness
2
Neither component carries domain know-how: the backbone's prior is broad but fixed, the harness controls procedure but not content
3
Without it, an agent loses budget within a task (inferring package behavior by trial and error) and fails to reuse what it learns across tasks
4
Operational knowledge is what turns knowing about a domain into being able to act in it — binding methods and tools to the problem
5
It has two complementary constituents: capability (invocable methods, code, APIs) and usage policy (when, why, and how to use each unit)
6
The research agent becomes $A_{\text{res}} = (M_\theta, H, K)$ , where $K$ is the operating context; the step policy becomes $a_t \sim \pi_\theta(a \mid \tau, h_t, H, K)$
7
The methodological challenge: produce operational knowledge automatically and at scale from declarative sources that drift with every release

Mathematical Formulation

The Research Task and Agent

$\tau = (q, D, E, g), \qquad A = (M_\theta, H) \;\longrightarrow\; A_{\text{res}} = (M_\theta, H, K)$

A task couples a problem q, its data D, an environment E (tools plus budget), and a target g. The standard agent is a backbone plus a harness; the research agent adds K, an explicit operating context carrying the operational knowledge the other two components never supply.

The Agent Loop with Operating Context

$a_t \sim \pi_\theta\big(a \mid \tau, h_t, H, K\big), \qquad o_t = E(a_t)$

At step t the agent conditions on the task, the history h_t, the harness H, and now the operating context K, acts, and observes the environment's response. K enters the conditioning of every action rather than adding a new control loop.

Key Points

1
$K$ is instantiated as a set of skills $K = \{S_1, \ldots, S_m\}$ — so producing operational knowledge reduces to producing skills
2
Adding a skill requires no change to $M_\theta$ or $H$ ; it becomes operating context that ports across compatible harnesses and accumulates across tasks
3
Each skill has three layers: $S = (\text{SKILL.md},\; \text{references/},\; \text{scripts/})$ — knowledge interface, knowledge substrate, execution interface
4
SKILL.md is the only layer read up front: goals, key concepts, tool usage, worked examples, failure modes, and routes to deeper material
5
references/ (API docs, algorithmic detail, configs) and scripts/ (executable wrappers) load only on demand — the progressive disclosure principle
6
A source's skills form a graph $\mathcal{G} = (S, L)$ with an entry skill and links encoding routing, dependency, or composition relations
7
Progressive disclosure over the graph means $K$ holds only the branch the task needs — an agent can carry thousands of skills yet read only a few

Mathematical Formulation

Operating Context as a Skill Set

$K = \{S_1, \ldots, S_m\}$

The operating context is a set of skills held at a given moment. Producing operational knowledge becomes the concrete problem of producing and selecting the right skills.

The Three-Layer Skill

$S = \big(\, \text{SKILL.md},\;\; \text{references/},\;\; \text{scripts/} \,\big)$

SKILL.md is the knowledge interface read up front; references/ is the knowledge substrate loaded on demand; scripts/ is the execution interface. The first turns capability into usage policy; the latter two turn knowledge into capability.

The Skill Graph

$\mathcal{G} = (S, L), \qquad S = \{S_i\}_{i=1}^{n},\; n \ge 1, \qquad L \subseteq \{(S_i, S_j) \in S \times S \mid i \neq j\}$

A source is distilled into a graph of skills S connected by links L. Each link is a routing, dependency, or composition relation; L may be empty when a source yields a single skill or several independent ones.

Key Points

1
Every distillation run — whatever triggers it — follows one four-stage loop: $z \to Q \to X \to \tilde{\mathcal{G}} \to (\mathcal{G}, R)$
2
Scope picks capabilities $Q$ ; ground gathers evidence $X$ ; construct assembles a candidate graph $\tilde{\mathcal{G}}$ ; verify yields an accepted graph $\mathcal{G}$ with a construction record $R$
3
Task-agnostic distillation anchors on a source $z = c$ (repo, paper, tutorial) and asks what it makes possible — long-lived skills built ahead of time
4
Task-oriented distillation anchors on a problem $z = \tau$ and asks what solving it demands — evidence is discovered and assembled, not handed over
5
The record $R$ retains the evidence used, checks performed, and unresolved gaps — skills carry their own provenance and known limits
6
Verification separates distillation from summarization: candidates are checked with assertion-backed cases, repo-native tests, CLI checks, tiny-fixture checks, or smoke scripts
7
A failure attributed to the graph triggers local repair and reruns; surviving gaps are recorded, never hidden

Mathematical Formulation

The Four-Stage Distillation Pipeline

$z \;\xrightarrow{\text{scope}}\; Q \;\xrightarrow{\text{ground}}\; X \;\xrightarrow{\text{construct}}\; \tilde{\mathcal{G}} \;\xrightarrow{\text{verify}}\; (\mathcal{G}, R)$

From an anchor z, scoping decides the capabilities Q, grounding gathers supporting evidence X, construction assembles the candidate graph in the three skill layers, and verification produces the accepted graph plus a construction record R.

The Two Anchors

$\text{task-agnostic: } z = c \in C \qquad\qquad \text{task-oriented: } z = \tau$

Task-agnostic runs anchor on a source c drawn from the available material C and build reusable skills ahead of time. Task-oriented runs anchor on a problem and search for the material a specific task needs. The anchor is the only structural difference between the two forms.

Key Points

1
DisCo is one agent in two modes over the same $M_\theta$ and $H$ : creator (distills and deposits graphs) and researcher (solves tasks with $K$ from the library)
2
The AREX-Skill Library is the seam: creator mode writes accepted graphs, researcher mode retrieves a task-relevant branch
3
The modes are asymmetric in cost: creator is paid once per source and amortized; researcher pays only for what a task opens
4
This asymmetry is what makes the layer scalable — distillation runs offline at ecosystem breadth, and task-solving inherits the result without re-deriving it
5
Consumption is progressive disclosure: the agent reads a router/entry skill, opens only what it needs, and follows links to setup, evaluation, diagnosis, or repair skills on demand
6
The interface adds operating context, not a new control loop — the same skills serve any compatible harness, which keeps the knowledge layer independent of harness design

Mathematical Formulation

The Two Modes over One Agent

$\underbrace{c \;\to\; (\mathcal{G}, R)}_{\text{creator mode}} \qquad\longrightarrow\qquad \underbrace{A_{\text{res}} = (M_\theta, H, K)}_{\text{researcher mode}}$

Creator mode maps a source to an accepted graph and its record, deposited in the library. Researcher mode is the research agent drawing K from that library. The same backbone and harness serve both; only the mode changes.

Key Points

1
1,000 ML repositories selected on open-source visibility and practical use, spanning models, training/deployment systems, data/evaluation tools, and scientific software
2
Each repository is distilled at ~$40 with GPT-5.5 and GPT-5.6-sol (xhigh reasoning); the evidence boundary is the repo's source, docs, examples, tests, scripts, and config
3
The snapshot holds 5,353 skills across 1,000 graphs, organized into 20 areas and 178 capability families
4
2,209 area-to-family assignments with 700 repos in more than one family — memberships overlap because a repo can support several capabilities
5
A generated router narrows a request area → family → repository graph; the taxonomy is fixed before assignment, with stars/URLs excluded from the model input
6
Classification rejects weak matches — keyword-only, dependency-only, optional-integration, and example-only assignments are not accepted
7
Beyond repositories: 636 paper-derived skills from 153 papers, plus task-oriented graphs per benchmark (MLE-bench, FrontierCS, PassNet)

Mathematical Formulation

Library Coverage at Scale

$\underbrace{1{,}000}_{\text{repositories}} \;\longrightarrow\; \underbrace{5{,}353}_{\text{skills}} \;\;\text{routed over}\;\; \underbrace{20}_{\text{areas}} \times \underbrace{178}_{\text{capability families}}$

One thousand repository graphs yield 5,353 verified skills, indexed by a two-level taxonomy of 20 areas and 178 capability families. The router uses this index to load only the branch a task needs.

Key Points

1
Clean ablation: Codex harness, GPT-5.5 backbone, fixed budget — the only variable is the presence of DisCo-distilled skills, constructed on a separate budget before run time
2
MLE-bench (75 comps): 31.11% → 72.89% Any-Medal — +41.78 points, the +134.3% headline; also beats the best public baseline (Famou-Agent 2.0, 64.44%)
3
Gains hold across tiers (High: 13.33 → 62.22, a 366.8% relative jump) — the advantage grows with task difficulty
4
PaperBench (20 papers): 29.45% → 39.59% (+10.14, +34.4%), improving 18 of 20; biggest lifts on low-baseline tasks (rice 7.94 → 48.51, ftrl 1.50 → 17.17)
5
FrontierCS (188 tasks): 70.63 → 77.14 (+6.51, +9.22%), 74 improved / 66 unchanged, bootstrap CI [3.41, 9.83]
6
PassNet: AS Score 1.343 → 1.5313 (+14.0%), correctness 81.35% → 90.76%, failed samples 14 → 5 (64.3% fewer), past TorchInductor's 1.419
7
Every gain comes from adding operating context, not a stronger model or a new control loop

Mathematical Formulation

The MLE-bench Headline

$\frac{72.89 - 31.11}{31.11} \;\approx\; 1.343 \;=\; +134.3\%$

The relative improvement on MLE-bench Any-Medal: from 31.11% without skills to 72.89% with them, a 41.78-point absolute gain, under a matched backbone, harness, and execution budget.

Key Points

1
The advantage grows with difficulty: MLE-bench High tasks gain +48.89 points; FrontierCS tasks below 50 rise from a 19.43 to 45.99 mean, 30 crossing the 50 threshold
2
The largest relative lifts land on low-baseline tasks (PaperBench ftrl 11.4x, rice 6.1x), while already-moderate tasks gain little (all-in-one +1.77, bam +2.18)
3
Gains are not explained by extra compute: on FrontierCS Spearman's $\rho$ is 0.006 (tokens), 0.014 (steps), 0.015 (tool calls) — essentially uncorrelated with per-task gain
4
A better score-efficiency frontier: 4.47M tokens/task vs Claude Code's 14.72M and 13.85M, at a higher score — Pareto-dominant on score, tokens, steps, and tool calls
5
The honest caveat: 2 of 20 PaperBench tasks regress (sample-specific-masks 57.11 → 52.04, stay-on-topic 32.31 → 27.79), both above the no-skill average
6
Likely cause is retrieval-precision failure — retrieved skills distract from a narrow, idiosyncratic strategy the base agent would otherwise discover
7
Suggested fix: better routing or an explicit fallback to unguided reasoning when retrieved skills are a poor match

Go deeper

Study Plans

LLM Study Plan ML Study Plan All Papers Practice Problems