LoRA Trainable Parameter Savings

Problem Statement

Given the shapes of the weight matrices you intend to adapt, a LoRA rank and a layer count, report how many parameters LoRA trains, how many full fine-tuning would train, and what percentage that is.

Background

LoRA freezes a pretrained weight W of shape (d_out, d_in) and learns a low-rank update instead:

$W' = W + \frac{\alpha}{r} BA, \qquad B \in \mathbb{R}^{d_{out} \times r},\; A \in \mathbb{R}^{r \times d_{in}}$

So the trainable count per adapted matrix is

$r \cdot d_{in} + r \cdot d_{out} = r\,(d_{in} + d_{out})$

against d_out * d_in for full fine-tuning. The scaling factor alpha/r is a constant applied at forward time - it adds no parameters, and neither A nor B carries a bias.

The saving is dramatic because the ratio is r(d_in + d_out) / (d_in * d_out). For a 4096x4096 projection at r = 8 that is 8 * 8192 / 16777216, about 0.39%. Multiply by the number of transformer layers you target and by how many matrices per layer you adapt (typically q_proj and v_proj) and the whole adapter for a 7B model still fits in a few megabytes - which is what makes it possible to fine-tune a VLM on a single consumer GPU and to serve dozens of task adapters over one shared base model.

Your Task

Implement:

def lora_stats(layer_shapes, rank, num_layers=1):

layer_shapes is a list of [d_out, d_in] pairs listing the matrices adapted inside one layer; that set repeats num_layers times.

Return [lora_params, full_params, percent] where the first two are ints and percent is 100 * lora_params / full_params rounded to 4 decimals.

Input Format

• layer_shapes - list of [d_out, d_in] integer pairs
• rank - positive integer r
• num_layers - positive integer

Output Format

[int, int, float]

Sample

print(lora_stats([[4096, 4096], [4096, 4096]], 8, 32))

Output:

[4194304, 1073741824, 0.3906]