Chapter 12: Production: Observability, Benchmarks, Runtimes & Eval-Driven Development

An agent that works in your dev environment is not a production agent. Production means: you can measure it, you can improve it iteratively from real failure data, you can debug a single user's broken trajectory, and you can ship a new version without regressing the old one. The discipline is eval-driven development — every production change starts with an eval, ends with measurement against that eval, and ships only when it improves the metric.

This chapter is the practitioner's stack:

• Benchmarks to measure capability
• Tracing standards (OTel GenAI) to capture per-trajectory data
• Observability platforms to query, inspect, and debug
• Runtime patterns for serving agents at scale
• Eval-driven development as the meta-loop that consumes all of the above

This chapter covers:

• SWE-bench & GAIA — the dominant coding/general-task agent benchmarks
• WebArena & OSWorld — web and OS interaction benchmarks
• OTel GenAI conventions — the standard for tracing LLM/agent runs
• LangSmith / LangFuse / Arize — the leading observability platforms
• Production runtimes — what changes when you serve at scale
• Eval-driven development — the workflow that ties measurement into shipping