Chapter 5: Memory I: Virtual Context & MemGPT
How agents remember things that don't fit in the context window. The context-window problem and why long-context models don't solve it. MemGPT's OS-inspired virtual memory, memory blocks, sleep-time compute for offline consolidation, and eviction/promotion policies.
Chapter Overview
An agent's only memory is its prompt. Once the conversation, tool results, and reflections grow past the context window, something has to give — and what gives determines whether the agent feels coherent or feels like a goldfish.
Long-context models (200k, 1M, 2M tokens) help but don't solve the problem. Three reasons: (1) cost scales linearly or worse with context length, (2) attention degrades — models perform worse on facts in the middle of long contexts (the 'lost in the middle' phenomenon), (3) some agent state genuinely outlives any single context window (a personal assistant that you've used for two years).
MemGPT (Packer et al., 2023, now productionized as Letta) reframed this: borrow the operating-systems trick of virtual memory. Treat the LLM context window as 'main memory' (small, fast) and an external store as 'disk' (large, slow). Page items in and out as needed. The model itself decides what to evict and what to recall, using tools.
This chapter covers:
- The context-window problem — why long context isn't a complete fix
- MemGPT: virtual context — paging items between context and external store
- Memory blocks architecture — typed segments (system, user, working, episodic) with explicit policies
- Sleep-time compute — offline consolidation when the agent is idle
- Evicting & promoting memories — what stays in core, what falls back to disk
Chapter Roadmap
Click any topic to jump in
Context Window Problem
Long context buys headroom but doesn't replace memory. Cost, lost-in-the-middle, and persistence all argue for external storage.
How items move (paging) and how they're shaped (blocks)
MemGPT: Virtual Context
OS-inspired virtual memory for LLMs — main context + archival, with the model itself paging items in and out.
Memory Blocks
Typed segments inside core context: system, persona, human, working — each with its own size and update policy.
Sleep-Time Compute
Offline consolidation when the agent is idle. Memory summarization, reflection generation, embedding updates.
Eviction & Promotion
What stays in core, what falls back to archival. The heart of every memory architecture.
An agent's only memory is its prompt. Every technique from the earlier chapters — the ReAct transcript, the plans, the reflections, the tool observations — accumulates as text in the context window, and a window is finite. When it fills, something has to give, and what gives decides whether the agent feels coherent across a long task or resets like a goldfish every few turns. This topic frames that problem precisely, because the rest of the chapter is a set of answers to it. The tempting escape hatch is a bigger window — 200K, 1M, 2M tokens — and this topic explains why that only buys headroom, not a solution. Cost scales roughly linearly with context length, so a million-token prompt is untenable per turn at scale; attention quality degrades in the 'lost in the middle' regime where facts buried mid-prompt are recalled worst; and some state genuinely outlives any window, like a personal assistant you have used for two years. We also separate the three flavors of memory — working, episodic, and semantic — that later topics map onto MemGPT's architecture, so 'memory' stops being one vague word.
Definition
The context-window problem is that an agent's entire memory is its finite prompt, so as history, tool results, and reflections accumulate, content must eventually be dropped or externalized. Larger context windows only defer it: cost scales with length, recall degrades for mid-prompt facts, and some state outlives any single window — external memory is unavoidable.
In this topic
Why Long Context Isn't Enough
The naive answer to memory is 'just use a 1M-token model.' Three reasons that fails:
- Cost. Each prompt's cost scales roughly linearly with context length. A 1M-token prompt at 5 dollars per million output tokens costs 5 dollars per turn. At 100 turns, 500 dollars per session — untenable for any consumer-facing product.
- Quality degradation. The 'lost in the middle' paper (Liu et al., 2023) showed that even strong long-context models score worst on facts buried in the middle of the prompt. Recall is bimodal: high at the start and end, much lower in the middle.
- Persistence. A personal assistant that remembers facts about you across years cannot fit two years of conversation in any context window. The fundamental need for external storage is unavoidable.
Long context buys you headroom. It doesn't replace memory architecture.
What 'Memory' Actually Means for Agents
Three distinct flavors of agent memory, often conflated:
- Working memory. The active context window — what the agent is reasoning about now. ~10K to ~200K tokens depending on the model.
- Episodic memory. Records of past sessions/conversations — 'last week we discussed X.' Often months or years deep. Not in working memory by default; pulled in on demand.
- Semantic memory. Facts the agent knows about the user/world — 'this user prefers French; their daughter's name is Marie.' Smaller than episodic, included in working memory if relevant.
MemGPT's architecture maps neatly: working memory is core context; semantic memory lives in a user block always loaded in-context; episodic memory lives in external storage and is retrieved by tool calls.
A user tells the agent: 'My daughter Marie is allergic to peanuts.' Three months later, the user asks: 'What snacks should I send to Marie's school?' How does each memory type contribute?