TinyLoRA

Standard LoRA decomposes weight updates into low-rank matrices $\Delta W = BA$ where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$. For a Qwen2.5-7B model with hidden dimension $d = k = 4096$, even at the minimum rank $r = 1$, each adapted module requires $d + k = 8{,}192$ parameters. At rank $r = 8$ (a common default), that's $r(d + k) = 65{,}536$ parameters per module — and with 560 modules across 80 transformer blocks (7 modules each: Q, K, V, O projections + 3 MLP matrices), the total reaches 36.7 million trainable parameters. This is efficient compared to full fine-tuning ($\sim$7B), but still represents enormous capacity.

LoRA-XS improves this by computing the truncated SVD of frozen weights $W = U\Sigma V^\top$ at rank $r$, then training only a small $r \times r$ matrix $R$ in the middle: $\Delta W = U\Sigma R V^\top$. At rank $r = 2$, this requires only $r^2 = 4$ trainable parameters per module — a massive reduction to $4 \times 560 = 2{,}240$ total parameters.

The fundamental limitation remains: both methods need at least one trainable parameter per module. The minimum total is proportional to $n_m$, the number of adapted modules. With 560 modules, even one parameter per module means 560 parameters — and each module's adapter is independent, preventing information sharing across the model. This architectural constraint makes sub-thousand parameter adaptation seem impossible.

TinyLoRA replaces the per-module trainable matrix $R$ with a globally shared low-dimensional vector $\mathbf{v} \in \mathbb{R}^u$ projected through fixed random tensors:

$W' = W + U\Sigma\left(\sum_{i=1}^{u} v_i P_i\right) V^\top$

Where:

• $W \in \mathbb{R}^{d \times k}$ is the frozen pretrained weight matrix (e.g., $4096 \times 4096$ for attention projections)
• $U \in \mathbb{R}^{d \times r}$, $\Sigma \in \mathbb{R}^{r \times r}$, $V \in \mathbb{R}^{k \times r}$ come from the truncated SVD of $W$ at rank $r = 2$
• $P_i \in \mathbb{R}^{r \times r}$ are fixed random matrices drawn once at initialization and never trained — each module gets its own unique set of $P_i$
• $\mathbf{v} \in \mathbb{R}^u$ is the only trainable component — as few as $u = 13$ values shared across all 560 modules

The mechanism works in three stages. First, the vector $\mathbf{v}$ is combined with the fixed random matrices to form a module-specific update matrix: $R_m = \sum_{i=1}^{u} v_i P_i^{(m)}$ where each module $m$ has its own unique random bases $P_i^{(m)}$. Second, this $r \times r$ matrix is sandwiched between the frozen SVD factors to produce the full-rank update: $\Delta W_m = U_m \Sigma_m R_m V_m^\top$. Third, the update is added to the frozen weights.

The random projections $P_i$ act as a fixed basis for the space of possible $r \times r$ weight update matrices. Training only needs to find the right linear combination coefficients $v_i$. This is justified by the Johnson-Lindenstrauss lemma: random projections from $\mathbb{R}^u$ into $\mathbb{R}^{r \times r}$ approximately preserve pairwise distances with high probability, so the essential structure of the update space is maintained even at extreme compression. The SVD factors $U, \Sigma, V$ further constrain updates to lie in the principal subspace of the original weights, ensuring alignment with the model's learned representations.

Parameter comparison at $d = k = 4096$, 560 modules: