Chapter 6: Memory II: Hybrid Stores, Mem0 & Skill Libraries
Beyond MemGPT's two-tier model: hybrid memory architectures that fuse vector, graph, and lexical retrieval; the Mem0 architecture for production-scale agent memory; Voyager-style skill libraries that turn execution traces into reusable code.
Chapter Overview
MemGPT solved the capacity problem — how to store more than fits in the context window. Memory II is about the quality problem — when you have memory, how do you find the right piece, and how do you turn experience into reusable knowledge?
Two families of techniques:
- Hybrid retrieval. A single retrieval mode (vector similarity) misses important matches. Mem0 (the production memory framework) fuses vector, graph, and lexical search, then re-ranks. The win: ~26% better recall vs. vector-only, with lower context usage.
- Skill libraries. Voyager (Wang et al., 2023) showed that an agent can turn successful execution traces into named, reusable skills (functions in code). On novel tasks, the agent retrieves matching skills instead of re-deriving them — exponential speedup on familiar problem classes.
This chapter covers:
- Hybrid vector + graph + lexical memory — when each retrieval mode wins
- Mem0 architecture — production-grade agent memory
- Voyager skill libraries — turning experience into code
- Skill acquisition vs retrieval — when to learn a new skill, when to use an old one
- Memory evaluation — how to measure whether your memory system is actually helping
Chapter Roadmap
Click any topic to jump in
Hybrid Retrieval
Vector + graph + lexical retrieval, fused via RRF. Each mode wins different queries.
Declarative facts (Mem0) and procedural skills (Voyager)
Mem0
Production memory framework: atomic facts, relationships, summaries — hybrid retrieval at all three levels.
Voyager Skills
Persistent procedural memory — turn successful execution traces into named, retrievable skills.
Acquisition vs Retrieval
When to learn a new skill, when to use an old one. The library-as-codebase discipline.
Memory Evaluation
Recall@k, consistency, latency, and the 'did memory help?' test that matters most.
Vector retrieval was the default for agent memory in 2023 — embed every memory, search by cosine similarity. It works for many cases but misses three: (a) facts that share entities but use different words, (b) exact matches buried under semantically similar noise, and (c) chains of relationships that no single embedding captures. Hybrid memory combines vector, graph, and lexical (BM25-style) retrieval and learns to weight them per query.
In this topic
What Each Retrieval Mode Is Good At
Vector retrieval. Embeds memory and query into a shared dense space; returns top- by cosine similarity. Wins when: the user's question and the relevant memory share concepts but not words ('the doctor I saw last month' → memory about Dr. Lee). Loses when: the user wants an exact identifier ('order #1234') — a vector store may return 'similar' orders rather than the exact match.
Graph retrieval. Stores memory as a knowledge graph: entities (people, projects, topics) connected by typed relationships ('works_with', 'mentioned_in'). Traverses from a query entity outward by relation. Wins when: the question is multi-hop ('what did Alice say about Bob's project?'). Loses when: the query is unstructured natural language with no clear entities.
Lexical retrieval (BM25 / keyword). Classic IR — score by term frequency × inverse document frequency. Wins when: the user uses specific keywords (acronyms, identifiers, proper nouns). Loses when: the user's terminology differs from what's in memory.
No single mode wins all queries. Hybrid retrieval runs all three in parallel and fuses the results.
Reciprocal Rank Fusion (RRF)
The dominant fusion method: Reciprocal Rank Fusion (RRF). For each retriever, get the top- results in ranked order. For each candidate document, sum across all retrievers (with a constant to dampen the head). The fused ranking is by the summed score.
RRF's appeal: scale-free. It does not require calibrated scores from each retriever (which is hard — vector cosines and BM25 scores live on different scales). Just the rank matters. A document that's #1 in vector AND #3 in BM25 is almost certainly the best answer. A document that's only #1 in vector but #50 in BM25 is suspect.
Production systems weight the retrievers (vector_weight = 0.6, lexical_weight = 0.3, graph_weight = 0.1) — typically learned from a retrieval-quality eval set.