Bootstrap Confidence Interval Width for Eval Accuracy

mediumPROProduction: Cost, Caching & EvalsAI AgentsAI Agents 1

Problem Statement

Report the normal-approximation 95% confidence interval for an agent's eval accuracy, so a small benchmark's noise is visible.

Background

For n graded items with k correct, the sample accuracy is p = k/n. The Wald 95% interval is *p +/- 1.96 * sqrt(p(1-p)/n)**, clamped to [0, 1]. Report the lower bound, upper bound, and width.

Your Task

def acc_ci(k, n):

Return a dict {"low": float, "high": float, "width": float}, each rounded to 4 decimals.

Input Format

k (int correct), n (int total, >= 1).

Output Format

A dict of three floats.

Sample

print(acc_ci(80, 100))

Output:

{'low': 0.7216, 'high': 0.8784, 'width': 0.1568}

Example

Input

print(acc_ci(80, 100))

Output

{'low': 0.7216, 'high': 0.8784, 'width': 0.1568}

Reasoning

Calculate the sample accuracy $p$ by dividing the number of correct items by the total items: $p = 80 / 100 = 0.8$ .
Compute the standard error of the proportion using the formula $\sqrt{\frac{p(1-p)}{n}}$ : $\sqrt{\frac{0.8 \times 0.2}{100}} = \sqrt{0.0016} = 0.04$ .
Determine the margin of error for a 95% confidence level by multiplying the standard error by the Z-score $1.96$ : $1.96 \times 0.04 = 0.0784$ .
Calculate the raw lower and upper bounds by subtracting and adding the margin of error to the sample accuracy: $\text{low} = 0.8 - 0.0784 = 0.7216$ and $\text{high} = 0.8 + 0.0784 = 0.8784$ .
Compute the width of the interval as the difference between the upper and lower bounds: $0.8784 - 0.7216 = 0.1568$ .
The final output is {'low': 0.7216, 'high': 0.8784, 'width': 0.1568}

Constraints

p = k/n; margin = 1.96*sqrt(p*(1-p)/n).
Clamp low/high to [0,1]; width = high - low.
Round each to 4 decimals.

Learn this in

Production: Observability, Benchmarks, Runtimes & Eval-Driven DevelopmentAgent Engineering Study Plan Probability & StatisticsFoundations

More from AI Agents 1

Cheapest Cache TTL Meeting a Freshness SLAhard Route to the Cheapest Model Meeting Qualitymedium pass@k Estimatormedium Savings from a Semantic Cachemedium Cache Hit Rateeasy

Concept primerPREVIEW

1. Background Knowledge

The Wald confidence interval is the most common normal-approximation interval for a population proportion. When you observe $k$ successes out of $n$ independent Bernoulli trials, the sample proportion is $\hat{p} = k/n$ . By the Central Limit Theorem, for large $n$ , the sampling distribution of $\hat{p}$ is approximately normal with mean $p$ and standard error $\sqrt{\hat{p}(1-\hat{p})/n}$ . A 95% confidence interval is then constructed as:

\hat{p} \pm z_{0.975} \cdot \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}

where $z_{0.975} \approx 1.96$ is the 97.5th percentile of the standard normal distribution. This interval captures the range within which we are 95% confident the true accuracy lies.

In the context of AI agent evaluation, a small benchmark (e.g., 100 items) has substantial statistical noise. Reporting only a point estimate like "80% accuracy" hides the uncertainty. The width of the confidence interval ( $\text{high} - \text{low}$ ) quantifies this noise: a wide interval means the benchmark is too small to distinguish between, say, 70% and 90% true accuracy. This is critical for production decisions about whether to trust an eval score or need more test items.

Note that the Wald interval can produce bounds outside $[0, 1]$ when $\hat{p}$ is near 0 or 1, so the result must be clamped to the valid probability range.

2. Algorithm Approach

This is a direct formula evaluation problem — no iterative or search-based algorithm is needed. The approach is:

Compute the sample proportion $\hat{p} = k/n$ .
Compute the standard error (SE) of the proportion.
Compute the margin of error as $1.96 \times \text{SE}$ .
Derive lower and upper bounds, clamping to $[0, 1]$ .
Compute the width as $\text{high} - \text{low}$ .
Round all three values to 4 decimal places and return them in a dictionary.

3. Step-by-Step Strategy

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results