Chapter 9: Specialized Agents: Computer Use & Voice
Two specialized agent modalities — agents that drive a screen by clicking and typing, and agents that converse over voice in real time. Covers Anthropic Computer Use, screen-grounding action grammars, Pipecat-based voice pipelines, LiveKit streaming, and the latency budgets that make real-time agents possible.
- Topics
- 5
- Videos
- 3
Chapter Overview
Most agents in this course exchange text with tool APIs. Two important specializations break that mold: computer-use agents that interact with arbitrary GUIs by clicking and typing, and voice agents that converse with users over phone or audio in real time. Each requires fundamentally different architecture — different action grammars, different latency budgets, different evaluation.
This chapter covers:
- Computer-use agents — Anthropic's Computer Use, OpenAI's Operator: how an agent sees a screen and acts on it
- Screen grounding & action grammars — what 'click' means when the model can only output text
- Voice agents (Pipecat) — building blocks for STT → LLM → TTS pipelines
- LiveKit streaming pipelines — production audio infra
- Latency budgets for realtime agents — the 800-millisecond ceiling and how to stay under it
Chapter Roadmap
Click any topic to jump in
Computer Use
Agents that drive a screen by clicking and typing — Anthropic Computer Use, OpenAI Operator.
Visual GUI agents and audio voice agents
Screen Grounding
Pixel-coordinate vs accessibility-tree grammars — and the hybrid most production systems use.
Voice Agents (Pipecat)
Frame-processor pipelines: STT → LLM → TTS, streaming end to end.
LiveKit
WebRTC transport for voice — sub-300ms latency, full-duplex, barge-in support.
Latency Budgets
The 800ms ceiling, allocated stage by stage. Where to optimize when you're over.
Computer Use Agents
Screen Grounding and Action Grammars
Voice Agents with Pipecat
Sign up to unlock this chapter
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.