Chapter 9: Specialized Agents: Computer Use & Voice
Two specialized agent modalities — agents that drive a screen by clicking and typing, and agents that converse over voice in real time. Covers Anthropic Computer Use, screen-grounding action grammars, Pipecat-based voice pipelines, LiveKit streaming, and the latency budgets that make real-time agents possible.
Chapter Overview
Most agents in this course exchange text with tool APIs. Two important specializations break that mold: computer-use agents that interact with arbitrary GUIs by clicking and typing, and voice agents that converse with users over phone or audio in real time. Each requires fundamentally different architecture — different action grammars, different latency budgets, different evaluation.
This chapter covers:
- Computer-use agents — Anthropic's Computer Use, OpenAI's Operator: how an agent sees a screen and acts on it
- Screen grounding & action grammars — what 'click' means when the model can only output text
- Voice agents (Pipecat) — building blocks for STT → LLM → TTS pipelines
- LiveKit streaming pipelines — production audio infra
- Latency budgets for realtime agents — the 800-millisecond ceiling and how to stay under it
Chapter Roadmap
Click any topic to jump in
Computer Use
Agents that drive a screen by clicking and typing — Anthropic Computer Use, OpenAI Operator.
Visual GUI agents and audio voice agents
Screen Grounding
Pixel-coordinate vs accessibility-tree grammars — and the hybrid most production systems use.
Voice Agents (Pipecat)
Frame-processor pipelines: STT → LLM → TTS, streaming end to end.
LiveKit
WebRTC transport for voice — sub-300ms latency, full-duplex, barge-in support.
Latency Budgets
The 800ms ceiling, allocated stage by stage. Where to optimize when you're over.
Computer-use agents can drive arbitrary GUIs by taking screenshots and emitting mouse/keyboard actions. Anthropic's Computer Use (Claude 3.5 Sonnet, Oct 2024) and OpenAI's Operator (early 2025) are the two production exemplars. The model literally sees pixel buffers and decides 'click at (450, 320)' or 'type "hello"'.
In this topic
The Computer-Use Loop
Each step of a computer-use agent looks like:
- Capture screenshot. A virtual display (Xvfb in a Linux VM, or an actual desktop) renders the current state. The screenshot is converted to a PNG and added to the model's context.
- Model emits an action. Either a mouse action (left_click, right_click, mouse_move), a keyboard action (type, key), or a 'tool' action (screenshot, wait).
- Controller executes. The action is dispatched to the virtual display via xdotool or pyautogui equivalents.
- Wait for re-render. Wait some milliseconds (or watch for screen changes), then go back to step 1.
This loop is expensive. Each step is a full LLM call with a 1080p screenshot in the context. P50 latency per action is 2-5 seconds. Computer-use agents are best for tasks that don't have an API — filing forms in legacy software, navigating SaaS dashboards without exposed APIs, automating desktop apps.
Failure Modes Specific to Computer-Use
Three failure modes that don't appear in API-based agents:
- Click misalignment. The model emits 'click at (450, 320)' but actually wants to click a button at (455, 318). With high-resolution screenshots, off-by-5-pixel errors are common. Mitigation: zoom-and-click — model first identifies the region, then zooms in for a refined click.
- Modal dialogs. A 'Cookies?' dialog appears mid-task. The model wasn't expecting it and gets confused. Mitigation: pre-prompt the model to expect interrupts and dismiss them.
- Race conditions. The model clicks, but the page hasn't loaded; the click goes to whatever was rendered before. Mitigation: explicit wait_for_screenshot_change action with a timeout.