Chapter 5: Fine-tuning

Transfer learning works because the pretrained loss landscape has already converged to a region with useful features. Formally, fine-tuning from pretrained weights $\theta_0$ finds $\theta^* = \arg\min_\theta \mathcal{L}_{\text{task}}(\theta)$ starting from $\theta_0$ instead of random initialization. The Hessian $H = \nabla^2 \mathcal{L}$ at $\theta_0$ has most eigenvalues near zero (the model has learned a low-dimensional manifold), so fine-tuning primarily moves along a small number of important directions. This is why fine-tuning converges in 3-5 epochs while pretraining takes hundreds of passes over the data.

Catastrophic forgetting occurs when fine-tuning gradients $\nabla \mathcal{L}_{\text{task}}$ are large in directions that are important for pretrained capabilities. The EWC (Elastic Weight Consolidation) regularizer $\lambda \sum_i F_i(\theta_i - \theta_i^{\text{pre}})^2$ weights the penalty by the Fisher information $F_i = \mathbb{E}[(\partial \log P / \partial \theta_i)^2]$, which measures how much parameter $\theta_i$ matters for the pretrained distribution. Parameters with high Fisher information are 'important' for pretraining and penalized more strongly during fine-tuning.

Feature extraction uses the pretrained model as a fixed function $f_\theta: \mathcal{X} \to \mathbb{R}^d$ and only trains a head $g_\phi: \mathbb{R}^d \to \mathcal{Y}$ with $|\phi| \ll |\theta|$ parameters. The capacity of the combined model is limited by the representational power of $f_\theta$ — if the pretrained features don't capture task-specific information, no amount of head training will help. Fine-tuning has $|\theta| + |\phi|$ trainable parameters, giving strictly more capacity, but the sample complexity scales as $O(|\theta|/n)$ where $n$ is the number of fine-tuning examples — risking overfitting when $n$ is small.

Continued pretraining on domain text reduces the KL divergence $D_{\text{KL}}(P_{\text{domain}} \| P_\theta)$ between the domain's language distribution and the model's learned distribution. Without domain adaptation, the model must generalize from the pretraining distribution $P_{\text{general}}$ — the error is bounded by $\epsilon_{\text{task}} \leq \epsilon_{\text{best}} + d_{\mathcal{H}}(P_{\text{general}}, P_{\text{domain}})$ where $d_{\mathcal{H}}$ is the domain divergence. Continued pretraining directly reduces this divergence term, tightening the error bound before task-specific fine-tuning even begins.