Week 9-10

Chapter 9: Specialized Agents: Computer Use & Voice

Two specialized agent modalities — agents that drive a screen by clicking and typing, and agents that converse over voice in real time. Covers Anthropic Computer Use, screen-grounding action grammars, Pipecat-based voice pipelines, LiveKit streaming, and the latency budgets that make real-time agents possible.

Chapter Overview

Most agents in this course exchange text with tool APIs. Two important specializations break that mold: computer-use agents that interact with arbitrary GUIs by clicking and typing, and voice agents that converse with users over phone or audio in real time. Each requires fundamentally different architecture — different action grammars, different latency budgets, different evaluation.

This chapter covers:

Computer-use agents — Anthropic's Computer Use, OpenAI's Operator: how an agent sees a screen and acts on it
Screen grounding & action grammars — what 'click' means when the model can only output text
Voice agents (Pipecat) — building blocks for STT → LLM → TTS pipelines
LiveKit streaming pipelines — production audio infra
Latency budgets for realtime agents — the 800-millisecond ceiling and how to stay under it

Chapter Roadmap

Click any topic to jump in

Computer Use

Agents that drive a screen by clicking and typing — Anthropic Computer Use, OpenAI Operator.

The Computer-Use LoopFailure Modes Specific to Computer-Use

Two specialized modalities with their own action models

Visual GUI agents and audio voice agents

Screen Grounding

Pixel-coordinate vs accessibility-tree grammars — and the hybrid most production systems use.

Pixel-Coordinate GrammarsAccessibility-Tree Grammars

Voice Agents (Pipecat)

Frame-processor pipelines: STT → LLM → TTS, streaming end to end.

The STT → LLM → TTS PipelinePipecat's Frame Processor Model

What carries the audio bits in real time

LiveKit

WebRTC transport for voice — sub-300ms latency, full-duplex, barge-in support.

Why WebRTC for Voice AgentsBarge-In and Turn Detection

The constraint that ties it all together for production

Latency Budgets

The 800ms ceiling, allocated stage by stage. Where to optimize when you're over.

The 800ms Budget, AllocatedWhere to Optimize

Computer-use agents can drive arbitrary GUIs by taking screenshots and emitting mouse/keyboard actions. Anthropic's Computer Use (Claude 3.5 Sonnet, Oct 2024) and OpenAI's Operator (early 2025) are the two production exemplars. The model literally sees pixel buffers and decides 'click at (450, 320)' or 'type "hello"'.

In this topic

1The Computer-Use Loop

2Failure Modes Specific to Computer-Use

1 of 2

The Computer-Use Loop

Each step of a computer-use agent looks like:

Capture screenshot. A virtual display (Xvfb in a Linux VM, or an actual desktop) renders the current state. The screenshot is converted to a PNG and added to the model's context.
Model emits an action. Either a mouse action (left_click, right_click, mouse_move), a keyboard action (type, key), or a 'tool' action (screenshot, wait).
Controller executes. The action is dispatched to the virtual display via xdotool or pyautogui equivalents.
Wait for re-render. Wait some milliseconds (or watch for screen changes), then go back to step 1.

This loop is expensive. Each step is a full LLM call with a 1080p screenshot in the context. P50 latency per action is 2-5 seconds. Computer-use agents are best for tasks that don't have an API — filing forms in legacy software, navigating SaaS dashboards without exposed APIs, automating desktop apps.

2 of 2

Failure Modes Specific to Computer-Use

Three failure modes that don't appear in API-based agents:

Click misalignment. The model emits 'click at (450, 320)' but actually wants to click a button at (455, 318). With high-resolution screenshots, off-by-5-pixel errors are common. Mitigation: zoom-and-click — model first identifies the region, then zooms in for a refined click.
Modal dialogs. A 'Cookies?' dialog appears mid-task. The model wasn't expecting it and gets confused. Mitigation: pre-prompt the model to expect interrupts and dismiss them.
Race conditions. The model clicks, but the page hasn't loaded; the click goes to whatever was rendered before. Mitigation: explicit wait_for_screenshot_change action with a timeout.

Papers

WebArena: A Realistic Web Environment for Building Autonomous Agents OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Blogs

Anthropic Computer Use OpenAI Operator