Chapter 6: Memory II: Hybrid Stores, Mem0 & Skill Libraries
Beyond MemGPT's two-tier model: hybrid memory architectures that fuse vector, graph, and lexical retrieval; the Mem0 architecture for production-scale agent memory; Voyager-style skill libraries that turn execution traces into reusable code.
- Topics
- 5
- Videos
- 3
- Min read
- 14
Chapter Overview
MemGPT solved the capacity problem — how to store more than fits in the context window. Memory II is about the quality problem — when you have memory, how do you find the right piece, and how do you turn experience into reusable knowledge?
Two families of techniques:
- Hybrid retrieval. A single retrieval mode (vector similarity) misses important matches. Mem0 (the production memory framework) fuses vector, graph, and lexical search, then re-ranks. The win: ~26% better recall vs. vector-only, with lower context usage.
- Skill libraries. Voyager (Wang et al., 2023) showed that an agent can turn successful execution traces into named, reusable skills (functions in code). On novel tasks, the agent retrieves matching skills instead of re-deriving them — exponential speedup on familiar problem classes.
This chapter covers:
- Hybrid vector + graph + lexical memory — when each retrieval mode wins
- Mem0 architecture — production-grade agent memory
- Voyager skill libraries — turning experience into code
- Skill acquisition vs retrieval — when to learn a new skill, when to use an old one
- Memory evaluation — how to measure whether your memory system is actually helping
Chapter Roadmap
Click any topic to jump in
Hybrid Retrieval
Vector + graph + lexical retrieval, fused via RRF. Each mode wins different queries.
Declarative facts (Mem0) and procedural skills (Voyager)
Mem0
Production memory framework: atomic facts, relationships, summaries — hybrid retrieval at all three levels.
Voyager Skills
Persistent procedural memory — turn successful execution traces into named, retrievable skills.
Acquisition vs Retrieval
When to learn a new skill, when to use an old one. The library-as-codebase discipline.
Memory Evaluation
Recall@k, consistency, latency, and the 'did memory help?' test that matters most.
Topics
5 topics in this chapter
Chapter 5 gave the agent somewhere to put its memories; it said nothing about how to find the right one again. The default answer in 2023 was a single move: embed every memory, embed the query, return the nearest vectors by cosine similarity. That works often enough to feel solved, which is exactly why its failures are dangerous. Vector search misses three whole classes of query — a fact that shares an entity with the query but none of its words, an exact identifier like an order number drowning under semantically similar neighbours, and a multi-hop question whose answer no single embedding ever captured. This topic is where memory retrieval stops being one algorithm and becomes a portfolio. You will meet the three retrieval modes — dense vector, graph traversal, and lexical BM25 — and see precisely which query each one wins and loses. Then the next topic, Reciprocal Rank Fusion, shows how to combine their rankings into one without needing their scores to live on the same scale. Hybrid retrieval is the foundation the rest of the chapter's production memory stack sits on.
Definition
Hybrid memory retrieval runs several complementary retrievers — typically dense vector search, knowledge-graph traversal, and lexical BM25 keyword matching — over the same memory store in parallel, then fuses their separate rankings into one. Each mode covers the queries the others miss, so the combined recall exceeds any single retriever, especially on entity-sharing, exact-identifier, and multi-hop questions.
In this topic
- 1What Each Retrieval Mode Is Good At
- 2Reciprocal Rank Fusion (RRF)
What Each Retrieval Mode Is Good At
Vector retrieval. Embeds memory and query into a shared dense space; returns top- by cosine similarity. Wins when: the user's question and the relevant memory share concepts but not words ('the doctor I saw last month' → memory about Dr. Lee). Loses when: the user wants an exact identifier ('order #1234') — a vector store may return 'similar' orders rather than the exact match.
Graph retrieval. Stores memory as a knowledge graph: entities (people, projects, topics) connected by typed relationships ('works_with', 'mentioned_in'). Traverses from a query entity outward by relation. Wins when: the question is multi-hop ('what did Alice say about Bob's project?'). Loses when: the query is unstructured natural language with no clear entities.
Lexical retrieval (BM25 / keyword). Classic IR — score by term frequency × inverse document frequency. Wins when: the user uses specific keywords (acronyms, identifiers, proper nouns). Loses when: the user's terminology differs from what's in memory.
No single mode wins all queries. Hybrid retrieval runs all three in parallel and fuses the results.
Reciprocal Rank Fusion (RRF)
The dominant fusion method: Reciprocal Rank Fusion (RRF). For each retriever, get the top- results in ranked order. For each candidate document, sum across all retrievers (with a constant to dampen the head). The fused ranking is by the summed score.
RRF's appeal: scale-free. It does not require calibrated scores from each retriever (which is hard — vector cosines and BM25 scores live on different scales). Just the rank matters. A document that's #1 in vector AND #3 in BM25 is almost certainly the best answer. A document that's only #1 in vector but #50 in BM25 is suspect.
Production systems weight the retrievers (vector_weight = 0.6, lexical_weight = 0.3, graph_weight = 0.1) — typically learned from a retrieval-quality eval set.
Related Problems on PixelBank
The previous topic gave you a retrieval portfolio; a production memory system needs the machinery around it — something to decide what becomes a memory in the first place, at what granularity, and how the pieces relate. Mem0 is that machinery, one of the leading production-grade agent-memory frameworks, and this topic dissects it. The problem it targets is scale over time: a two-year assistant relationship produces millions of tokens of conversation, almost all of it noise, and stuffing raw transcripts into a vector store retrieves poorly and costs enormously. Mem0's answer is to store memory at three levels of abstraction — atomic facts with provenance, a relationship graph built from those facts, and periodic prose summaries — and to run the hybrid retrieval you just learned across all three in parallel. Its headline benchmark: roughly 26% better recall than a vector-only baseline on long conversations while using about ten times less context. The two concepts here are the layered memory model itself and the fact-extraction pipeline that feeds it, because extraction quality, not the retrieval stack, is where Mem0's win actually comes from.
Definition
Mem0 is a production agent-memory framework that stores memory at three abstraction levels — atomic facts with provenance, a relationship graph linking them, and periodic prose summaries — and retrieves across all three with hybrid search. An LLM-driven extraction pipeline turns raw conversation into deduplicated facts, which is the main source of its recall advantage over vector-only stores.
In this topic
- 1The Mem0 Memory Model
- 2Fact Extraction: Where the Quality Comes From
The Mem0 Memory Model
Mem0 stores memories at three levels of abstraction:
- Atomic facts. 'User's name is Alex. User lives in Seattle. User's daughter Marie is 7.' Each fact is a small, self-contained statement with provenance metadata (when said, by whom, source).
- Relationships. A graph layer connecting facts: Alex --has_daughter--> Marie, Marie --is_age--> 7. Built incrementally from atomic facts via entity extraction.
- Summaries. Periodic distillations of related facts into prose: 'Alex is a software engineer in Seattle with a 7-year-old daughter, Marie, who is allergic to peanuts.' Used as a fast lookup when the agent needs broad context, not specific facts.
A query enters → retrieves at all three levels in parallel → RRF fusion → injection into context.
Fact Extraction: Where the Quality Comes From
The most important component of Mem0 is the fact extractor — the pipeline that turns raw conversation into atomic facts. Three steps:
- Entity recognition. Find entities (people, organizations, dates, products) in the conversation. NER models or LLM-as-extractor.
- Relation extraction. Find typed relationships between entities. 'Alex met Bob at Acme' → (Alex, met_at, Acme), (Bob, met_at, Acme), (Alex, met, Bob).
- Deduplication & resolution. A new fact arrives ('User's name is Alex Smith'). Already a fact 'User's name is Alex.' Merge: keep the more specific 'Alex Smith' and link to the same canonical entity.
Quality of extraction directly determines quality of retrieval. Mem0's contribution is doing this step well at scale, not the retrieval stack itself.
Related Problems on PixelBank
Mem0 and hybrid retrieval both store declarative memory: facts the agent knows. But an agent that solves a hard task and then forgets how it solved it pays the full cost again next time. The missing piece is procedural memory — remembered skill, not remembered fact — and Voyager is where this course makes it concrete. Voyager (Wang et al., 2023) was built to play Minecraft, but ignore the game; its lasting contribution is the skill library. Every time the agent accomplishes a task with code that verifiably works, it stores that code as a named, vector-indexed skill. Facing a new task, it first searches the library, and if a relevant skill exists it retrieves and extends it instead of writing fresh code. This topic builds the three moving parts — an automatic curriculum that proposes progressively harder tasks, the skill library itself, and the iterative prompt-execute-verify-save loop — and then shows the pattern transferring far beyond games, to coding, data-analysis, and support agents. The following topic asks the harder question this raises: when is a new experience actually worth storing as a skill?
Definition
A skill library is persistent procedural memory: whenever an agent completes a task with code that verifiably succeeds, it stores that code as a named, vector-indexed skill described in natural language. On later tasks the agent retrieves and adapts matching skills rather than re-deriving them, amortizing the one-time cost of discovery across every future reuse.
In this topic
- 1How Voyager Builds Skills
- 2Skill Libraries Beyond Games
How Voyager Builds Skills
Three components:
- Curriculum module. Suggests progressively harder tasks ('chop wood' → 'craft a pickaxe' → 'mine stone' → ...). Drives skill acquisition by exploring the task space.
- Skill library. A vector-indexed store where each entry is {name, description, code, examples, embedding}. Searched by task description.
- Iterative prompting. For a new task, retrieve the top- matching skills, attempt the task, run an executor to verify success, and (if successful) save the new code as a skill. If failed, generate a self-critique and retry.
In Minecraft, this scaled to hundreds of skills like place_torch, kill_zombie, find_diamond. The agent built a tech tree of competencies entirely from execution traces — no human curated the library.
Skill Libraries Beyond Games
The pattern transfers directly to:
- Coding agents. Each successfully passing test suite produces a function that gets named, vector-indexed, and reused. SWE-bench solvers like Claude Code use a variant of this for project-specific helpers.
- Data-analysis agents. Each Python notebook that solved a particular question (e.g., 'compute the year-over-year growth') becomes a callable skill. Future questions retrieve the matching skill and adapt.
- Customer-support agents. Successful resolution paths ('authenticate user → check policy → process refund → send confirmation email') become named workflows the agent can call by name.
The key win is amortization: the cost of figuring out how to do something is paid once. Future invocations only pay the cost of retrieval.
A coding agent handles 100 distinct refactor tasks per day for a team. After a month, the skill library has ~200 entries. How does this change the average task cost?
Voyager showed that an agent can turn its own successes into a growing library of skills. This topic confronts the discipline that keeps that library an asset rather than a liability, because a skill library is not helped by more skills — it is helped by the right skills. Acquire too freely and the library fills with near-duplicates and one-off answers that were never reusable, so retrieval slows and returns noise. Acquire too rarely and the agent keeps re-deriving work it has already done, defeating the entire point. The resolution is a similarity-threshold policy on every new task: a strong match reuses the existing skill untouched, a weak match adapts it and saves the adaptation only when it is genuinely different, and no match solves from scratch but stores the result only after a generalization check proves it works on more than the single task that produced it. The second concept covers the maintenance side — merging, promoting, demoting, and versioning — because a skill library is a codebase, and one that only grows and never refactors rots. The chapter's final topic then measures whether any of this is working.
Definition
Skill acquisition is the decision of whether a completed task should be stored as a new library skill; skill retrieval is reusing an existing one. A similarity threshold governs the choice — strong matches reuse, weak matches adapt, and novel tasks are stored only after passing a generalization check — preventing both duplicate bloat and wasteful re-derivation.
In this topic
- 1When to Acquire a New Skill
- 2Library Maintenance
When to Acquire a New Skill
A new task arrives. The agent retrieves the top- skills. Three cases:
- Strong match (similarity > 0.85). Just use the existing skill. Don't add a duplicate.
- Weak match (0.5–0.85). Adapt the existing skill — most likely the same parametric task with different inputs. Save the adapted version only if it's substantially different from the parent (not a hyperparameter tweak).
- No match (< 0.5). Solve from scratch and save as a new skill — but only after the task succeeds AND the skill passes a generalization check (does it work on at least one synthetic similar task?).
The generalization check guards against memorizing one-off tasks. A skill that solves ONE task is just a cached answer; a skill that solves a CLASS of tasks earns its place in the library.
Library Maintenance
A library that only grows turns into a heap. Production systems run periodic consolidation:
- Merge near-duplicates. Two skills with similarity > 0.92 → one canonical entry plus aliases.
- Promote frequently-used. Skills accessed >100 times go into a 'fast path' index that's checked first.
- Demote unused. Skills not accessed in 90 days move to cold storage; not deleted, just deprioritized in retrieval.
- Versioning. When a skill is patched (it now handles a new edge case), keep the old version available; the index points to the latest by default.
The analogy: a skill library is a codebase. Treat it like one — refactor, deprecate, document.
Every topic in this chapter added memory machinery — hybrid retrieval, Mem0's layered store, Voyager's skills, acquisition policies. This closing topic asks the question that should have been haunting all of them: how do you know any of it is actually helping? Memory subsystems are uniquely hard to evaluate because they affect agent quality only indirectly — the agent answers better because it happened to remember something — and the benefit is spread thinly across many turns rather than concentrated in one visible win. That indirection makes it easy to ship an expensive memory system that feels sophisticated and changes almost nothing. This topic gives the metrics that resist that trap. The first concept covers three quality dimensions you can track continuously — Recall@k against a curated query set, cross-session consistency of remembered facts, and retrieval latency at the tail. The second covers the single most important and most-skipped measurement: the counterfactual 'did memory help?' test, run by answering each turn twice, once with retrieved memory and once without, and counting how often memory changed the answer for the better. Without that number, every memory system feels good.
Definition
Memory evaluation measures whether an agent's memory subsystem improves its answers, despite the effect being indirect and diffuse. Core metrics are Recall@k against a curated query set, cross-session fact consistency, and retrieval latency, plus a counterfactual test that reruns each turn with and without retrieved memory to isolate memory's actual contribution.
In this topic
- 1Three Quality Dimensions
- 2The 'Did Memory Help?' Test
Three Quality Dimensions
1. Recall@k. Given a query that has a known correct memory, does the retriever return it in the top-? Measured against a curated eval set of (query, gold_memory) pairs. Standard IR metric, easy to track over time.
2. Cross-session consistency. Does the agent give the same answer to 'what's the user's name?' across 10 sessions, 1 month apart? Measure: rate of fact-changing-without-cause. A regression in this number means your eviction policy is too aggressive.
3. Latency. How long does retrieval add to the user-visible turn time? P50, P95, P99. A memory subsystem that adds 2 seconds to every turn is unlikely to be worth it, no matter how good the recall is.
The 'Did Memory Help?' Test
The most important question, and the easiest to skip: would the agent have answered correctly without the retrieved memory? Measure by running each turn twice — once with retrieved memory, once without. Compute: 'fraction of turns where memory changed the agent's answer for the better.'
For most production systems this number is 5–15%. Sounds low, but it's a multiplier on the baseline — and the cost of memory retrieval (typically <100ms) is small. Watch the number quarterly. If it's trending down, your memory store is full of low-value content.
Theory Exercise
Problem:
Your memory subsystem reports 80% recall@5 on the eval set, but users complain the agent 'feels forgetful.' What's likely going on?
Hints:
- Is the eval set representative of real queries?
- Could the issue be in promotion/eviction, not retrieval?
- Is it a quality issue or a latency issue?