Count Model Parameters

1. Background Knowledge

Neural network parameters are the learnable weights and biases in layers that are optimized during training. In PyTorch, nn.Linear(in_features, out_features) creates a layer with:

• Weight matrix: $$\mathbf{W} \in \mathbb{R}^{\text{out_features}$\times$\text{in_features}} → $$\text{in_features}$\times$\text{out_features}$$ parameters
• Bias vector: $$\mathbf{b} \in \mathbb{R}^{\text{out_features}} → $$\text{out_features}$$ parameters
• Total: $$\text{in_features}$\times$\text{out_features}$+$\text{out_features}$$

Trainable parameters have requires_grad=True (default for nn.Linear). ReLU activations have 0 parameters as they are fixed functions: $f(x) = \max(0, x)$.

Key PyTorch methods:

for name, param in model.named_parameters():
 if param.requires_grad:
 count = param.numel() # Total elements in tensor

Sample calculation: For Linear(10→20) + Linear(20→5):

• Layer 1: $10 \times 20 + 20 = 220$
• Layer 2: $20 \times 5 + 5 = 105$
• Total: 325 trainable parameters

2. Algorithm Approach

Direct iteration over parameters (most reliable):

1. Build model: Sequential(Linear1 → ReLU → Linear2)
2. Iterate model.named_parameters()
3. For each param: if requires_grad, add param.numel()
4. Track layer-specific counts using name filtering

Mathematical formulation: Let $L_1 =$\text{Linear}$(n_{in}, n_h)$, $L_2 =$\text{Linear}$(n_h, n_{out})$

params(L_i) = n_{prev} × n_i + n_i
total_params = params(L_1) + params(L_2)

Alternative: sum(p.numel() for p in model.parameters() if p.requires_grad)

3. Step-by-Step Strategy

def count_parameters(in_features, hidden_features, out_features):
 # Step 1: Create model
 model = nn.Sequential(
 nn.Linear(in_features, hidden_features),
 nn.ReLU(),
 nn.Linear(hidden_features, out_features)
 )

 # Step 2: Initialize counters
 total = 0
 layer1 = 0
 layer2 = 0