Chapter 12: Production: Observability, Benchmarks, Runtimes & Eval-Driven Development
What it takes to operate agents at scale: the dominant agent benchmarks (SWE-bench, GAIA, WebArena, OSWorld), OpenTelemetry's GenAI semantic conventions, observability platforms (LangSmith, LangFuse, Arize), production runtime patterns, and eval-driven development as the discipline that ties it all together.
Chapter Overview
An agent that works in your dev environment is not a production agent. Production means: you can measure it, you can improve it iteratively from real failure data, you can debug a single user's broken trajectory, and you can ship a new version without regressing the old one. The discipline is eval-driven development — every production change starts with an eval, ends with measurement against that eval, and ships only when it improves the metric.
This chapter is the practitioner's stack:
- Benchmarks to measure capability
- Tracing standards (OTel GenAI) to capture per-trajectory data
- Observability platforms to query, inspect, and debug
- Runtime patterns for serving agents at scale
- Eval-driven development as the meta-loop that consumes all of the above
This chapter covers:
- SWE-bench & GAIA — the dominant coding/general-task agent benchmarks
- WebArena & OSWorld — web and OS interaction benchmarks
- OTel GenAI conventions — the standard for tracing LLM/agent runs
- LangSmith / LangFuse / Arize — the leading observability platforms
- Production runtimes — what changes when you serve at scale
- Eval-driven development — the workflow that ties measurement into shipping
Chapter Roadmap
Click any topic to jump in
SWE-Bench & GAIA
The dominant benchmarks for coding agents (SWE-Bench) and general assistants (GAIA).
WebArena & OSWorld
Interactive benchmarks for browser and OS-driving agents.
Standard schema and the platforms that consume it
OTel GenAI
The standard for tracing LLM/agent runs — backend-portable, cost-aware, joinable with app telemetry.
Observability Platforms
LangSmith, LangFuse, Arize — pick on workflow fit; all speak OTel.
Production Runtimes
Sync request-response for short jobs, async queue + checkpoints for long ones.
Eval-Driven Dev
The loop that consumes everything else: trace → cluster → eval → ship → repeat.
Two benchmarks that anchor the conversation about what 'a good agent' means in 2025.
SWE-bench measures coding agents on real GitHub issues from popular Python repos. GAIA measures general-task assistants on multi-step problems requiring reasoning, tool use, and grounding. Numbers on these two benchmarks set the public goalposts for the field.
In this topic
SWE-bench — Real GitHub Issues
SWE-bench (Jimenez et al., 2023) gives an agent a real GitHub issue from one of 12 popular Python repos (Django, scikit-learn, requests, etc.) plus the repo state at the time the issue was opened. The agent must produce a patch. The patch is judged by whether it makes the repo's test suite pass on a held-out test that was added to fix this issue.
Key numbers (SWE-bench Verified, the cleaned-up subset):
- GPT-4 baseline (no agent loop): ~2%
- Claude 3.5 Sonnet with a custom agent: ~50%
- Claude Opus 4 with the SWE-Agent harness: ~70%+
The benchmark is hard because it requires reading large unfamiliar codebases, finding the right file to change, writing a patch that passes hidden tests, and not breaking any prior tests. It's the closest public proxy for 'can the agent do real software engineering.'
GAIA — General Tasks
GAIA (Mialon et al., 2023) is a benchmark of 466 multi-step questions designed to require reasoning, tool use, and grounding (not just memorization). Examples:
- 'Of the men's tennis players in the top 10 of the ATP rankings on March 14, 2014, how many had won a Grand Slam?'
- 'What is the maximum value pictured in the chart in the SECOND-to-last issue of Computers in Industry, 2005?'
These require web search, document reading, image understanding, arithmetic, and judgment about whether the answer is verified. GAIA's three difficulty levels (Level 1: easy, web search; Level 2: medium, multi-tool; Level 3: hard, multi-step + judgment).
State-of-the-art on GAIA Level 1 in early 2025 is ~75% (frontier agents). Humans average ~92% on Level 1 with a web browser. The gap on Level 3 is much wider — humans ~80%, agents ~30-40%.