Loading...
Learning to Reason in 13 Parameters
John X. Morris, Niloofar Mireshghallah, Mark Ibrahim, Saeed Mahloujifar
Read the Paper on arXivHow many parameters does it take to teach a language model to reason? The conventional answer has been millions — but TinyLoRA challenges this assumption dramatically.
Standard LoRA fine-tuning of a 7B model at rank introduces roughly 40M trainable parameters — low-rank matrices and across every attention and MLP projection. Even LoRA-XS, which trains only matrices within the SVD of frozen weights, still needs thousands of parameters. TinyLoRA pushes this to the extreme: it replaces the per-module trainable matrix with a single shared vector projected through fixed random tensors , achieving what the authors call "rank below one" adaptation.
The key result: a Qwen2.5-7B model (hidden dim , 80 transformer blocks, 560 adapted modules) achieves 91% accuracy on GSM8K with only 13 trained parameters — stored in just 26 bytes of bf16. These 13 values are shared across all 560 module positions via weight tying, and each position produces a unique weight update because the fixed random projection matrices and the module-specific SVD factors are different at every position.
The paper reveals a surprising asymmetry: reinforcement learning (GRPO) is dramatically more parameter-efficient than supervised fine-tuning for reasoning tasks. While SFT at 13 parameters achieves only 83% (5% above the 78% base model baseline), RL achieves 91% — a 13-point gain. At 120 parameters, the gap widens further: RL reaches 95% while SFT remains at 84%. The explanation is rooted in signal density: SFT provides a dense loss signal over every token in a chain-of-thought solution (formatting, intermediate steps, final answer), overwhelming a 13-parameter budget. RL's binary reward (correct/incorrect final answer) correlates cleanly with the few features the tiny adapter can control.
An additional scaling law emerges: larger pretrained models require fewer TinyLoRA parameters to reach the same fraction of peak performance. To reach 95% of peak GSM8K accuracy, Qwen2.5-3B needs ~350 parameters, Qwen2.5-7B needs ~120, and Qwen2.5-14B needs even fewer. This suggests that reasoning capability is increasingly latent in larger models, waiting to be activated with a minimal nudge. If 13 parameters can unlock mathematical reasoning in a 7B model, what capabilities are already present in trillion-parameter models?
Click any topic to jump in
Projects a 13-dim vector through fixed random bases and frozen SVD factors to construct per-module weight updates.
Weight tying across 560 modules drops effective per-module rank to 0.023 — below any classical bound.
Binary RL reward is a 1-D signal that maps cleanly to tiny capacity; SFT's ~200-D token loss overwhelms it.
Group-relative policy optimization uses the group mean as baseline, no value network, producing a clean gradient.
Larger pretrained models need fewer adaptation parameters — reasoning is latent, adapter just activates it.
TinyLoRA extends parameter-efficient fine-tuning to extreme compression by projecting a tiny trainable vector through fixed random matrices, reducing the trainable parameter count from millions to as few as 13 while maintaining strong task performance.
Standard LoRA decomposes weight updates into low-rank matrices where and . For a Qwen2.5-7B model with hidden dimension , even at the minimum rank , each adapted module requires parameters. At rank (a common default), that's parameters per module — and with 560 modules across 80 transformer blocks (7 modules each: Q, K, V, O projections + 3 MLP matrices), the total reaches 36.7 million trainable parameters. This is efficient compared to full fine-tuning (7B), but still represents enormous capacity.
LoRA-XS improves this by computing the truncated SVD of frozen weights at rank , then training only a small matrix in the middle: . At rank , this requires only trainable parameters per module — a massive reduction to total parameters.
The fundamental limitation remains: both methods need at least one trainable parameter per module. The minimum total is proportional to , the number of adapted modules. With 560 modules, even one parameter per module means 560 parameters — and each module's adapter is independent, preventing information sharing across the model. This architectural constraint makes sub-thousand parameter adaptation seem impossible.
TinyLoRA replaces the per-module trainable matrix with a globally shared low-dimensional vector projected through fixed random tensors:
Where:
The mechanism works in three stages. First, the vector is combined with the fixed random matrices to form a module-specific update matrix: where each module has its own unique random bases . Second, this matrix is sandwiched between the frozen SVD factors to produce the full-rank update: . Third, the update is added to the frozen weights.
The random projections act as a fixed basis for the space of possible weight update matrices. Training only needs to find the right linear combination coefficients . This is justified by the Johnson-Lindenstrauss lemma: random projections from into approximately preserve pairwise distances with high probability, so the essential structure of the update space is maintained even at extreme compression. The SVD factors further constrain updates to lie in the principal subspace of the original weights, ensuring alignment with the model's learned representations.
Parameter comparison at , 560 modules:
| Method | Params per module | Total params |
|---|---|---|
| Full fine-tuning | 7B | |
| LoRA () | 36.7M | |
| LoRA () | 4.6M | |
| LoRA-XS () | 2,240 | |
| TinyLoRA (, full tying) | shared | 13 |
Replaces per-module trainable matrix with a single shared vector projected through unique fixed random matrices at each module
Random projections preserve the structure of the update space (Johnson-Lindenstrauss lemma) — distances in the -dimensional space map to distances in the matrix space
Frozen SVD factors constrain updates to lie in the principal subspace of the original weights, ensuring structural alignment with pretrained representations
Parameters reduced from (LoRA-XS) to total — a 172x reduction from LoRA-XS's 2,240 to TinyLoRA's 13
Each module produces a unique despite sharing the same , because the random and SVD factors differ per module — analogous to how DNA produces different proteins in different cellular contexts
Truncated SVD at rank works best — diminishing returns beyond because higher-rank components capture less variance in pretrained weights
The weight update at module m is constructed by linearly combining that module's fixed random matrices P_i with the shared trainable coefficients v_i, then projecting through that module's frozen SVD factors. Each module gets a unique ΔW from the same v.
u is the vector dimension (13), n_m is the number of adapted modules (560), and n_tie is the weight-tying factor. With full tying (n_tie = n_m = 560): total = u = 13. With no tying (n_tie = 1): total = u × n_m = 7,280.
The entire model adaptation — all weight updates across 560 modules in a 7B model — is encoded in 26 bytes. This fits in a single CPU cache line (64 bytes).
The Johnson–Lindenstrauss lemma guarantees that random linear projections from into approximately preserve pairwise distances. This means a tiny -dim vector retains the essential geometry of the full update space — training just finds the right linear combination of fixed random bases.