Repo-To-Skill

Ask an agent to improve a model on an unfamiliar dataset and it must decide which method suits the problem, which package implements it, how that package wants the data laid out, which configurations are appropriate, and which pitfalls silently invalidate an otherwise plausible run. It can infer all of this by trial and error — propose a plan, run it, inspect the failure, revise. The trouble is that those failures spend the same budget reserved for evaluation, and a single misconfigured run can burn a large share of it before producing any meaningful measurement.

The paper formalizes a research task as $\tau = (q, D, E, g)$ — a problem $q$, its data and material $D$, an environment $E$ that exposes tools and bounds a budget, and a target $g$ the result must meet — and the standard agentic system as $A = (M_\theta, H)$: an LLM backbone and a harness. The backbone supplies understanding, reasoning, planning, and execution; the harness supplies orchestration, memory, verification, and iterative refinement. Almost all recent progress has come from strengthening one or the other. But for research tasks, the two-component view leaves something unspecified: the backbone's prior is broad but fixed, and the harness controls procedure but supplies no domain content.

That gap is what the authors call operational knowledge — the expertise that binds the field's methods and tools to the task at hand by specifying what can solve it, when each candidate applies, and how it should be used. They write the research agent as $A_{\text{res}} = (M_\theta, H, K)$, adding an explicit operating context $K$. Operational knowledge has two constituents that must travel together: one turns knowledge into capability (methods, code, models, and APIs packaged into things the agent can actually invoke), and the other turns capability into usage policy (the conditions, reasons, and procedures that govern each unit's use). Capability without policy is a tool with no selection criteria; policy without capability is advice with no executable interface.

Crucially, this is not the harness in disguise. The harness specializes how an agent explores; $K$ specializes what it knows to consider. And declarative sources — a paper reporting that a technique improves accuracy, a repo documenting what an API accepts — state what holds, never a course of action for a given problem. The central methodological problem is to derive operational knowledge from that declarative material automatically and at scale.