pass@k Scorecard for an Agent Eval Suite

Problem Statement

Agents are stochastic, so a single run per task tells you almost nothing - the same agent can solve a SWE-bench task on attempt three and fail it on attempts one and two. Eval-driven development needs a metric that accounts for that: pass@k, the probability that at least one of k sampled attempts succeeds.

Background

Taking the empirical rate of "at least one pass in the first k runs" is biased and noisy. The standard unbiased estimator instead uses the counts directly: for a task with n total runs of which c passed,

$\text{pass@}k = 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}}$

The numerator counts the ways to draw k runs that are all failures. When c = 0 this is 1, giving pass@k = 0; when n - c < k there is no all-failure draw, so the binomial is 0 and pass@k = 1.

If a task has fewer than k runs, evaluate it at k' = n instead of discarding it.

A task where 0 < c < n is flaky - it neither reliably passes nor reliably fails, and those are the tasks worth reading traces for.

Your Task

Implement:

def scorecard(runs, k):

• runs: list of {"task_id": str, "tag": str, "passed": bool}, possibly many runs per task. A task's tag is taken from its first run in the list.

Return:

• tasks: number of distinct tasks
• pass_at_k: the unweighted mean of the per-task pass@k over all tasks, rounded to 4 decimal places (0.0 when there are no tasks)
• by_tag: dict of tag -> mean per-task pass@k for that tag, rounded to 4 dp, built by iterating tags in sorted order
• flaky: sorted list of task ids with 0 < c < n

Input/Output Format

Returns the four-key dict. Use math.comb; every averaged float is rounded to 4 dp.

Sample

runs = ([{"task_id": "t1", "tag": "web", "passed": True}]
        + [{"task_id": "t1", "tag": "web", "passed": False}] * 3)
print(scorecard(runs, 2)["pass_at_k"])
# 0.5

One pass in four runs: 1 - C(3,2)/C(4,2) = 1 - 3/6 = 0.5.