Chapter 5: Memory I: Virtual Context & MemGPT
How agents remember things that don't fit in the context window. The context-window problem and why long-context models don't solve it. MemGPT's OS-inspired virtual memory, memory blocks, sleep-time compute for offline consolidation, and eviction/promotion policies.
- Topics
- 5
- Videos
- 3
- Min read
- 16
Chapter Overview
An agent's only memory is its prompt. Once the conversation, tool results, and reflections grow past the context window, something has to give — and what gives determines whether the agent feels coherent or feels like a goldfish.
Long-context models (200k, 1M, 2M tokens) help but don't solve the problem. Three reasons: (1) cost scales linearly or worse with context length, (2) attention degrades — models perform worse on facts in the middle of long contexts (the 'lost in the middle' phenomenon), (3) some agent state genuinely outlives any single context window (a personal assistant that you've used for two years).
MemGPT (Packer et al., 2023, now productionized as Letta) reframed this: borrow the operating-systems trick of virtual memory. Treat the LLM context window as 'main memory' (small, fast) and an external store as 'disk' (large, slow). Page items in and out as needed. The model itself decides what to evict and what to recall, using tools.
This chapter covers:
- The context-window problem — why long context isn't a complete fix
- MemGPT: virtual context — paging items between context and external store
- Memory blocks architecture — typed segments (system, user, working, episodic) with explicit policies
- Sleep-time compute — offline consolidation when the agent is idle
- Evicting & promoting memories — what stays in core, what falls back to disk
Chapter Roadmap
Click any topic to jump in
Context Window Problem
Long context buys headroom but doesn't replace memory. Cost, lost-in-the-middle, and persistence all argue for external storage.
How items move (paging) and how they're shaped (blocks)
MemGPT: Virtual Context
OS-inspired virtual memory for LLMs — main context + archival, with the model itself paging items in and out.
Memory Blocks
Typed segments inside core context: system, persona, human, working — each with its own size and update policy.
Sleep-Time Compute
Offline consolidation when the agent is idle. Memory summarization, reflection generation, embedding updates.
Eviction & Promotion
What stays in core, what falls back to archival. The heart of every memory architecture.
Topics
5 topics in this chapter
An agent's only memory is its prompt. Every technique from the earlier chapters — the ReAct transcript, the plans, the reflections, the tool observations — accumulates as text in the context window, and a window is finite. When it fills, something has to give, and what gives decides whether the agent feels coherent across a long task or resets like a goldfish every few turns. This topic frames that problem precisely, because the rest of the chapter is a set of answers to it. The tempting escape hatch is a bigger window — 200K, 1M, 2M tokens — and this topic explains why that only buys headroom, not a solution. Cost scales roughly linearly with context length, so a million-token prompt is untenable per turn at scale; attention quality degrades in the 'lost in the middle' regime where facts buried mid-prompt are recalled worst; and some state genuinely outlives any window, like a personal assistant you have used for two years. We also separate the three flavors of memory — working, episodic, and semantic — that later topics map onto MemGPT's architecture, so 'memory' stops being one vague word.
Definition
The context-window problem is that an agent's entire memory is its finite prompt, so as history, tool results, and reflections accumulate, content must eventually be dropped or externalized. Larger context windows only defer it: cost scales with length, recall degrades for mid-prompt facts, and some state outlives any single window — external memory is unavoidable.
In this topic
- 1Why Long Context Isn't Enough
- 2What 'Memory' Actually Means for Agents
Why Long Context Isn't Enough
The naive answer to memory is 'just use a 1M-token model.' Three reasons that fails:
- Cost. Each prompt's cost scales roughly linearly with context length. A 1M-token prompt at 5 dollars per million output tokens costs 5 dollars per turn. At 100 turns, 500 dollars per session — untenable for any consumer-facing product.
- Quality degradation. The 'lost in the middle' paper (Liu et al., 2023) showed that even strong long-context models score worst on facts buried in the middle of the prompt. Recall is bimodal: high at the start and end, much lower in the middle.
- Persistence. A personal assistant that remembers facts about you across years cannot fit two years of conversation in any context window. The fundamental need for external storage is unavoidable.
Long context buys you headroom. It doesn't replace memory architecture.
What 'Memory' Actually Means for Agents
Three distinct flavors of agent memory, often conflated:
- Working memory. The active context window — what the agent is reasoning about now. ~10K to ~200K tokens depending on the model.
- Episodic memory. Records of past sessions/conversations — 'last week we discussed X.' Often months or years deep. Not in working memory by default; pulled in on demand.
- Semantic memory. Facts the agent knows about the user/world — 'this user prefers French; their daughter's name is Marie.' Smaller than episodic, included in working memory if relevant.
MemGPT's architecture maps neatly: working memory is core context; semantic memory lives in a user block always loaded in-context; episodic memory lives in external storage and is retrieved by tool calls.
A user tells the agent: 'My daughter Marie is allergic to peanuts.' Three months later, the user asks: 'What snacks should I send to Marie's school?' How does each memory type contribute?
The previous topic established that agents need external storage; this one introduces the architecture that made that idea concrete and practical. MemGPT (Packer et al., 2023, now productionized as Letta) borrows the oldest trick in operating systems — virtual memory — and applies it to the LLM. Treat the context window as fast 'main memory' (small, always visible) and an external store as slow 'disk' (large, archival), then page items between them as the task demands. The decisive design choice is who does the paging: MemGPT gives the model itself new tools — core_memory_append, core_memory_replace, archival_memory_insert, archival_memory_search — so the agent decides what to remember, what to overwrite, and what to recall. This topic builds the two-tier architecture and the 'function-call heartbeat' loop, where a single user-visible turn may hide several internal hops as the agent writes and retrieves memory before replying. Compaction — summarizing the oldest turns, archiving the originals, and keeping only the summary in context — is what keeps the working set bounded no matter how long the conversation runs.
Definition
MemGPT applies operating-system virtual memory to an LLM: the context window is fast main memory and an external store is archival 'disk', with the model paging data between them via memory tools it calls itself. Compaction summarizes and archives old turns to keep the working set within the window, letting the agent decide what to retain and retrieve.
In this topic
- 1The Two-Tier Architecture
- 2The 'Function Call Heartbeat' Loop
The Two-Tier Architecture
MemGPT splits memory into two tiers:
- Main context (in the prompt): always loaded, always visible to the LLM. Capped at the context-window size minus reserved space for the conversation. Holds: system prompt, core memory blocks (typed segments, see next topic), and recent conversation.
- Archival memory (external store): unbounded. Holds: every observation, every prior turn, every fact the agent decided was 'worth keeping but not in the working set.' Backed by a vector store with embeddings for semantic search.
The LLM has tools to move data between the tiers. When the conversation grows past the context budget, MemGPT triggers a compaction: summarize the oldest N turns into a brief paragraph, write the full transcript to archival memory, and replace the in-context turns with the summary. The summary lives in working memory; the originals are retrievable by archival_memory_search when needed.
The 'Function Call Heartbeat' Loop
The MemGPT loop is structured around the model's choice to either speak to the user or invoke a memory tool:
- New user message arrives → controller assembles the prompt (system + core memory + recent turns + user message).
- LLM generates either:
- send_message_to_user(content) — visible reply
- core_memory_append(block, content) — write a fact to a typed block
- core_memory_replace(block, content) — overwrite a block (e.g., user's location changed)
- archival_memory_insert(content) — store in long-term archive
- archival_memory_search(query, top_k) — pull older content back
- Controller dispatches the chosen tool, then re-prompts the LLM with the result.
- Loop continues until the LLM emits send_message_to_user.
This means a single 'turn' from the user's perspective may include several internal hops where the agent updates its memory, retrieves prior context, etc., before producing its reply.
MemGPT's main context is not one undifferentiated blob of remembered text — that would reintroduce the very 'vector-soup' failure Chapter 1 warned about. Instead it is partitioned into typed memory blocks, each with a distinct role, size limit, and update policy, and designing those blocks well is most of the real work in building a long-running agent. This topic walks Letta's reference block types: a fixed system_instructions block, a rarely-changing persona block that holds the agent's identity, a frequently-updated human block for what the agent knows about the user, and a high-churn working block that is scratch space for the current task. The payoff of typing memory by role is retrieval without search: 'what is the user's name?' goes straight to the human block, 'what was I just doing?' to the working block, no embeddings involved. We also cover the size-tuning knob — each block has a token budget, overflow ships to archival, and telemetry on observed block sizes beats guessing — because a block that constantly overflows and one that never fills are both misconfigured.
Definition
A memory-blocks architecture partitions an agent's core context into typed segments — such as system, persona, human, and working — each with its own size limit and update policy. Typing memory by role gives predictable, search-free retrieval and bounded size, with per-block overflow consolidated or shipped to archival storage rather than growing without limit.
In this topic
- 1The Standard Block Types
- 2Block Size Tuning
The Standard Block Types
Letta's reference architecture defines four block types:
- system_instructions block. Constant. Loaded into every prompt. Holds the system role description. Updated only by the developer.
- persona block. The agent's stable identity ('You are Quincy, a friendly customer-support agent for Acme Corp.'). Updated rarely.
- human (or user) block. What the agent knows about the current user. Updated frequently as facts accumulate ('user prefers French; user's company is Acme; user's main project is Phoenix').
- working block. Scratch space for the current task. Updated every turn ('current task: drafting refund reply for order #1234; collected facts: ...').
Each block has a configured size limit (in tokens). The model uses core_memory_append and core_memory_replace to update them. When a block hits its size limit, the model is prompted to consolidate — typically rewriting the block more concisely, with overflow content shipped to archival memory.
The key win: by structuring memory by role, the model knows where to look for what. 'What's the user's name?' → human block. 'What was I just doing?' → working block. No vector search needed for the common case.
Why is putting the user's name in a dedicated human block better than dropping it into a flat 'system memory' string?
Block Size Tuning
Block sizes are a design knob. A reasonable starting point for a personal assistant:
- system_instructions: 800 tokens (fixed)
- persona: 200 tokens (almost fixed)
- human: 2000 tokens
- working: 1000 tokens
- Reserved for current conversation: 4000 tokens
- Total core context: ~8000 tokens, leaving headroom in any modern model
If your human block keeps overflowing, increase its size — but you'll pay more per prompt. If it never fills up, decrease it. Telemetry: log the size of each block at each turn; tune from observed distributions, not guesses.
Every topic so far assumed the agent works only while a user is waiting on it. But an agent spends most of its life idle — between turns, overnight, between conversations — and that idle time is a resource. Sleep-time compute is the catch-all term for the useful work an agent does offline, without a user prompt driving it, and this topic surveys what is worth doing and when it pays off. Three activities dominate: memory consolidation, where recent episodic entries are grouped by topic and summarized into more durable form, an agent analog of REM sleep; reflection generation, where the day's failures become Reflexion-style notes for future use, kept off the latency-critical path; and embedding-index updates, where newly ingested content is embedded offline so long ingests never block a user. The governing trade-off is honest: sleep-time compute is real LLM spend during 'free' time, justified only when the value of the consolidated state exceeds its cost — true for a personal assistant used daily, false for a one-shot agent. Recent work even argues offline compute can partly substitute for online model size.
Definition
Sleep-time compute is offline work an agent performs while idle, without a user prompt: consolidating and summarizing recent memories, generating reflections from past failures, and updating embedding indexes. It trades real LLM spend during idle time for higher-quality accumulated state, worthwhile when that state's value to a recurring user exceeds the per-call cost.
In this topic
- 1What to Do During Sleep
- 2The Cost-Quality Frontier of Sleep-Time
What to Do During Sleep
Three high-value sleep-time activities:
- Memory consolidation. Take the last episodic memory entries, group them by topic, and write summaries. Like REM sleep for the agent — compress experience into more durable form.
- Reflection generation. Look at the last day's failures and produce reflections (Reflexion-style) for future use. Doing this offline rather than synchronously after each failure means the user-visible loop stays fast.
- Embedding index updates. New content ingested today gets embedded and added to the vector store. If you do this synchronously, every long ingest blocks the user; offline you can amortize the cost.
The trade-off: sleep-time compute is real cost (LLM calls during 'idle' time). Worth it when the value of the consolidated state exceeds the per-call cost — typically true for personal assistants used daily, not true for one-shot agents.
The Cost-Quality Frontier of Sleep-Time
Recent work (sleep-time compute, Inference scaling — DeepMind 2024) makes the case that offline inference compute can be a substitute for online parameters. A small model that does extensive sleep-time consolidation can rival a large model that does no consolidation, on tasks that benefit from accumulated experience.
For the practitioner: budget some fraction of your total LLM spend to sleep-time activities (1-5% is a reasonable starting point). Measure whether agent quality improves over time. If yes, scale the budget; if no, you don't need it for your use case.
Related Problems on PixelBank
The blocks and tiers of the previous topics raise one unavoidable question: memory is finite, so what stays and what goes? Every fact held in core context crowds out another, and every memory shipped to archival is one the agent might later fail to retrieve. This closing topic is about the two policies that resolve that tension and sit at the heart of any memory system. Eviction policies decide what to drop from core — the classic operating-system menu of LRU, LFU, explicit priority, and recency-weighted similarity, usually combined so system and user blocks are never evicted while the working block uses LRU. Promotion policies decide the reverse: when to pull an archival memory back into core, triggered either by an explicit search this turn or by recurring access, where a fact retrieved repeatedly earns a permanent seat in the user block. That second mechanism is what makes a long-running assistant feel coherent — facts that keep coming up become 'obvious' and need no retrieval, while new facts stay in archival until they prove they matter.
Definition
Eviction policies govern what an agent drops from finite core context — LRU, LFU, explicit priority, or recency-weighted similarity, typically combined by block. Promotion policies govern the reverse, pulling archival memories back into core on explicit retrieval or after recurring access. Together they decide what stays resident and what falls back to external storage.
In this topic
- 1Eviction Policies
- 2Promotion Policies
Eviction Policies
The standard OS-derived options:
- LRU (Least Recently Used). Drop the memory item that's been touched longest ago. Default for many systems. Works well when access patterns are bursty (you talk about Project A for a week, then Project B).
- LFU (Least Frequently Used). Drop the memory accessed least often. Useful when some items are 'one-hit wonders' (a fact that came up once and never again).
- Priority-based. Each memory has an explicit priority. The user's name is high priority — never evict. A passing remark is low priority — evict freely.
- Recency-weighted similarity. A hybrid: when the model wants to retrieve, score by cosine_similarity * decay(age). Old facts can still surface if highly relevant, but recent facts have an advantage.
Most production systems combine policies: priority for system/user blocks (no eviction); LRU for working block; recency-weighted similarity for archival retrieval.
Promotion Policies
The opposite question: when to pull an archival memory back into core. Two triggers:
- Explicit retrieval. The model uses archival_memory_search(query) and the result is injected into working memory for this turn. After the turn, it falls back to archival unless promoted.
- Recurring access. If the agent has retrieved the same memory 3+ times in recent conversations, it's a strong signal the memory should live in core. Auto-promote it from archival into the user block.
The second is what makes a long-running personal assistant feel coherent: facts that come up repeatedly become 'obvious' to the agent, no retrieval needed. New facts live in archival until they prove themselves.
An agent has been talking to a user for two months. Initially the user mentioned 'I work at Acme.' The agent stored this in archival memory. By month 2, the user has said 'Acme' 50+ times in conversation. What should happen?
Theory Exercise
Problem:
Your agent's user block keeps overflowing. You can either (a) increase the block size, (b) be more aggressive about evicting old facts, or (c) promote less from archival. Pick one and explain.
Hints:
- What's the cost of each option?
- Where does the bloat come from?
- Could the right answer be a combination?