Composing Continual Learning Mechanisms
Continual Learning Mechanisms Compose for Long-Horizon Memorization
Zheyuan Zhang, Alvin Zhang, Daniel Khashabi, Tianmin Shu
Read the Paper on arXivPaper Overview
Composing Continual Learning Mechanisms (Zhang, Zhang, Khashabi & Shu — Johns Hopkins University, 2026) asks a deceptively simple question: if a language model updates its own parameters to absorb new information, can it still remember what it learned a hundred updates ago? The authors call this long-horizon memorization, and they study it through continual supervised fine-tuning on a stream of 100 query-answer tasks — no stored examples from earlier tasks, no task identifier at inference. The model must learn each new task while holding onto every association it has already been taught.
The starting fact is bleak. After 100 tasks, naive sequential fine-tuning retains only 1.2% of what it learned. Catastrophic forgetting at this horizon is near-total, and — crucially — no single continual-learning mechanism the authors evaluate fixes it. Every standalone method (replay, distillation, EWC, LoRA tricks) is eliminated well before the 100-task mark.
The paper's thesis is that mechanisms addressing complementary sources of forgetting should work far better when composed than any of them alone. To make that testable, the authors organize the design space along two axes. Anchors answer what prior information each update should preserve: a data anchor (generative replay), a function anchor (self-distillation), and a weight anchor (importance-based regularization). Low-rank allocation rules answer where successive updates are retained: shared LoRA reuses one adapter, merged LoRA folds each update into the dense weights and reinitializes. Crossing these gives a large combinatorial space, which they navigate with task-level successive halving and then dissect with a 2⁴ factorial experiment.
The payoff is a clean composition result. The best method — all three anchors plus merged LoRA — is the only configuration that ranks among the top 3 on all three datasets, and it raises average final retention from 1.2% to 34.9%, a 28-fold improvement. The two biggest levers, the data anchor and merged LoRA, interact super-additively: together they buy far more than the sum of their parts.
Chapter Roadmap
Click any topic to jump in
Long-Horizon Memorization
Learn 100 query-answer tasks by continual fine-tuning, no stored examples, no task IDs — and naive SFT forgets almost everything, retaining just 1.2%.
Three Anchors
Data, function, and weight anchors each specify WHAT prior information an update must preserve — replay, self-distillation, importance-weighted regularization.
Low-Rank Allocation
Shared vs merged LoRA decides WHERE each update is retained — reuse one adapter, or fold it into the dense weights and start a fresh one every task.
Searching the Design Space
Task-level successive halving prunes 90 compositions down to 10 over longer horizons; a 2⁴ factorial then measures each mechanism's main and interaction effects.
Results
The best method combines all three anchors with merged LoRA — top 3 on every dataset, 34.9% average final retention, a 28-fold gain over naive fine-tuning.
What Composes, and What Doesn't
Replay and merged LoRA are the super-additive core; SI and SD help conditionally; and memory half-life is delayed, not defeated.
Prompting and retrieval can hand a model new information at inference time, but that information lives outside the parameters and has to be supplied again on every call. This paper asks the opposite question: can repeated parameter updates build and preserve memory inside the model itself? For parameters to serve as memory, the model must still recall an association after many more updates have landed on top of it.
The setting is continual supervised fine-tuning in the domain-incremental regime of Van de Ven and Tolias. A stream of tasks arrives one at a time. Task contains query-answer pairs ; when it arrives the model inherits the previous state but may not revisit raw examples from earlier tasks, and at inference it is never told which task a query belongs to. Because the study is about memorization rather than generalization, each task is evaluated on exactly the examples it was trained on — the question is pure recall, not transfer.
The obstacle is catastrophic forgetting: an update that teaches task overwrites parameters that encoded earlier associations. The tension the whole paper lives inside is the stability-plasticity trade-off — stay plastic enough to learn the new task, stay stable enough not to erase the old ones. At a 100-task horizon this is brutal. Naive fine-tuning has a memory half-life of a single task on two of the three datasets: an association is more likely than not to be gone by the very next update. Final retention after all 100 tasks sits at 1.2%.
What makes long-horizon memorization a distinct continual-learning problem — and not just a rerun of image-classification benchmarks — is the combination of a long horizon, no task identity, no rehearsal buffer of real data, and a recall (rather than transfer) objective. Existing language benchmarks mostly measure transfer across heterogeneous downstream tasks or changing corpora; sequential model-editing studies targeted edits. None isolates this question, so the authors build the evaluation from scratch.
Key Points
Long-horizon memorization: learn 100 query-answer tasks by continual SFT, retaining associations across all of them — memory inside the parameters, not supplied at inference like prompting or retrieval
Domain-incremental setting: no task identifier is given at inference, and raw examples from earlier tasks may not be revisited
It is memorization, not generalization — each task is evaluated on the same examples it was trained on, so the metric is pure recall
Catastrophic forgetting is the core obstacle: each new update overwrites parameters that stored earlier associations (the stability-plasticity trade-off)
Naive sequential fine-tuning retains only 1.2% after 100 tasks, with a memory half-life of ~1 task on Symbol-QA and LLM-QA
No single mechanism the authors evaluate maintains strong retention at this horizon — which motivates composing complementary mechanisms
Mathematical Formulation
Current-Task SFT Loss
The standard likelihood objective for task . Query tokens are not masked — the authors argue it is hard to separate query from answer in applications like test-time training, so the whole sequence is scored. This term drives plasticity; the anchors that follow supply stability.