Loading...
What it takes to operate agents at scale: the dominant agent benchmarks (SWE-bench, GAIA, WebArena, OSWorld), OpenTelemetry's GenAI semantic conventions, observability platforms (LangSmith, LangFuse, Arize), production runtime patterns, and eval-driven development as the discipline that ties it all together.
An agent that works in your dev environment is not a production agent. Production means: you can measure it, you can improve it iteratively from real failure data, you can debug a single user's broken trajectory, and you can ship a new version without regressing the old one. The discipline is eval-driven development — every production change starts with an eval, ends with measurement against that eval, and ships only when it improves the metric.
This chapter is the practitioner's stack:
This chapter covers:
Click any topic to jump in
The dominant benchmarks for coding agents (SWE-Bench) and general assistants (GAIA).
Interactive benchmarks for browser and OS-driving agents.
Standard schema and the platforms that consume it
The standard for tracing LLM/agent runs — backend-portable, cost-aware, joinable with app telemetry.
LangSmith, LangFuse, Arize — pick on workflow fit; all speak OTel.
Sync request-response for short jobs, async queue + checkpoints for long ones.
The loop that consumes everything else: trace → cluster → eval → ship → repeat.
Two benchmarks that anchor the conversation about what 'a good agent' means in 2025.
SWE-bench measures coding agents on real GitHub issues from popular Python repos. GAIA measures general-task assistants on multi-step problems requiring reasoning, tool use, and grounding. Numbers on these two benchmarks set the public goalposts for the field.
SWE-bench (Jimenez et al., 2023) gives an agent a real GitHub issue from one of 12 popular Python repos (Django, scikit-learn, requests, etc.) plus the repo state at the time the issue was opened. The agent must produce a patch. The patch is judged by whether it makes the repo's test suite pass on a held-out test that was added to fix this issue.
Key numbers (SWE-bench Verified, the cleaned-up subset):
The benchmark is hard because it requires reading large unfamiliar codebases, finding the right file to change, writing a patch that passes hidden tests, and not breaking any prior tests. It's the closest public proxy for 'can the agent do real software engineering.'
GAIA (Mialon et al., 2023) is a benchmark of 466 multi-step questions designed to require reasoning, tool use, and grounding (not just memorization). Examples:
These require web search, document reading, image understanding, arithmetic, and judgment about whether the answer is verified. GAIA's three difficulty levels (Level 1: easy, web search; Level 2: medium, multi-tool; Level 3: hard, multi-step + judgment).
State-of-the-art on GAIA Level 1 in early 2025 is ~75% (frontier agents). Humans average ~92% on Level 1 with a web browser. The gap on Level 3 is much wider — humans ~80%, agents ~30-40%.