Deduplicate and Evict Agent Memories

Problem Statement

An agent that writes to memory after every turn accumulates near-duplicates: "User prefers dark mode.", "user prefers dark mode", "User prefers dark mode!". Left alone this becomes the "vector-DB soup" failure mode - retrieval returns three copies of the same fact and burns the context window on redundancy.

Build the consolidation pass that runs before memory is written back.

Background

Consolidation is two stages:

Stage 1 - deduplicate by normalised key. Two memories collide when their normalised text is identical. Normalisation is: lowercase, strip leading/trailing whitespace, remove the characters ., ,, !, ?, ;, :, ' and ", then collapse every run of whitespace to a single space. Within a collision group keep exactly one record - the newest (largest timestamp); ties go to the higher importance; still tied, the lexicographically smallest id.

Stage 2 - evict to capacity. Rank the survivors by importance descending, then timestamp descending, then id ascending, and keep the first capacity.

Note the ordering matters: dedup first, so that three copies of a low-value fact cannot crowd out a distinct high-value one.

Your Task

Implement:

def consolidate_memories(memories, capacity):

• memories: list of dicts {"id": str, "text": str, "timestamp": int, "importance": int}.
• capacity: maximum number of memories to retain.

Return the retained memory dicts in the Stage 2 ranking order (highest priority first).

Input/Output Format

Returns list[dict] - the original dict objects, at most capacity of them, ordered by (-importance, -timestamp, id).

Sample

mem = [
    {"id": "m1", "text": "User prefers dark mode.", "timestamp": 10, "importance": 3},
    {"id": "m2", "text": "  user prefers   DARK mode ", "timestamp": 20, "importance": 1},
    {"id": "m3", "text": "Deploys on Fridays", "timestamp": 5, "importance": 5},
]
print([m["id"] for m in consolidate_memories(mem, 5)])
# ['m3', 'm2']

m1 and m2 normalise to the same key user prefers dark mode; m2 is newer so it survives - even though m1 has higher importance.