Chapter 5: Fine-tuning

Week 5-6

Learn how to adapt pretrained LLMs to specific tasks and domains. Master the spectrum of fine-tuning approaches from full parameter updates to parameter-efficient methods like LoRA and QLoRA, understand instruction tuning that transforms base models into helpful assistants, and learn best practices for data preparation and evaluation.

Topics: 5
Videos: 5
Min read: 40

Chapter Overview

Pretraining gives an LLM broad language understanding, but a pretrained model is just a next-token predictor---it does not follow instructions, answer questions helpfully, or refuse harmful requests. Fine-tuning bridges this gap by adapting the model to specific behaviors or domains using curated datasets.

There are two major paradigms of fine-tuning. Full fine-tuning updates all model parameters, achieving the highest quality but requiring significant compute (often 10-100x less than pretraining, but still substantial for large models). Parameter-efficient fine-tuning (PEFT) methods like LoRA update only a small fraction of parameters, making it possible to fine-tune 70B+ models on a single GPU.

The most impactful form of fine-tuning for modern LLMs is instruction tuning: training on (instruction, response) pairs to make the model follow user requests. This is what transforms a base model like Llama into a chat model like Llama-Chat. Combined with RLHF (Chapter 6), instruction tuning produces the helpful, harmless assistants we interact with today.

This chapter covers:

Transfer Learning: The theoretical foundation for why fine-tuning works
Full Fine-tuning: When and how to update all parameters
LoRA & QLoRA: Parameter-efficient fine-tuning that democratized LLM adaptation
Instruction Tuning: Creating helpful assistants from base models
Data Preparation: Building high-quality fine-tuning datasets

Chapter Roadmap

Click any topic to jump in

Explainer video

Transfer Learning

Why pretrained knowledge transfers to downstream tasks — catastrophic forgetting, feature extraction, and domain adaptation.

Why Transfer Learning Works for LLMsCatastrophic ForgettingFeature Extraction vs Fine-TuningDomain Adaptation

Two approaches to adapting pretrained models

Full updates vs parameter-efficient methods

Full Fine-tuning

Updating all parameters with SFT — when maximum quality justifies the compute cost and overfitting risks.

Full Fine-Tuning SetupSupervised Fine-Tuning (SFT)Hyperparameter SensitivityWhen to Use Full Fine-Tuning

LoRA & QLoRA

Low-rank adaptation that trains <1% of parameters — the method that democratized fine-tuning on consumer GPUs.

LoRA: Low-Rank AdaptationWhy Low-Rank WorksQLoRA: Quantized LoRALoRA Hyperparameters

Applying fine-tuning in practice

Instruction formats and data quality

Instruction Tuning

Transforming base models into helpful assistants with instruction-response pairs and chat templates.

What is Instruction TuningInstruction Dataset DesignMulti-Task Instruction TuningChat Templates and Formatting

Data Preparation

Quality over quantity — decontamination, formatting consistency, and augmentation for fine-tuning datasets.

Data Quality Over QuantityFormatting and ConsistencyDecontaminationData Augmentation

Topics

5 topics in this chapter

A pretrained language model has read trillions of tokens, but it has never been asked to do your job. Training a new model for every task, such as radiology reports, contract review, or support tickets, would mean paying for pretraining again and again, usually with far too little labelled data to succeed. The practical question is how to reuse what the model already knows and change only what the new task needs, without breaking the rest.

The previous chapter, Pretraining, showed how next-token prediction on web-scale text builds that general knowledge. This chapter, Fine-tuning, covers how to adapt it, and this first topic explains why adaptation works at all.

We start with why transfer learning works for LLMs: the pretrained weights already encode grammar, facts, and reasoning, so a task needs only a small push. Then we cover catastrophic forgetting, the cost of pushing too hard. Next we compare feature extraction, which freezes the model, with fine-tuning, which updates it. We finish with domain adaptation, a middle stage of continued pretraining on in-domain text.

Definition

Transfer learning is the reuse of parameters learned on a source task as the starting point for a target task. For LLMs, the source task is next-token prediction on broad text and the target is a narrower task or domain. Training starts from the pretrained weights $\theta_0$ instead of a random initialisation, so far less target data and compute are needed.

In this topic

1Why Transfer Learning Works for LLMs
2Catastrophic Forgetting
3Feature Extraction vs Fine-Tuning
4Domain Adaptation

1 of 4

Why Transfer Learning Works for LLMs

To predict the next token well across the whole web, a model must learn spelling, grammar, word meaning, common facts, and simple reasoning, because all of them reduce its loss. Probing studies find a rough layering, with surface and syntactic features clearest in lower layers and more semantic, task-like features higher up. A new task mostly reuses these features and changes how they are combined. That is why fine-tuning needs thousands of examples instead of billions of tokens. ULMFiT showed that with 100 labelled examples, a fine-tuned language model matched training from scratch on 100 times more data. Transfer fails when the target domain shares little with pretraining text.

Mathematical Intuition

Transfer learning works because the pretrained loss landscape has already converged to a region with useful features. Formally, fine-tuning from pretrained weights $\theta_0$ finds $\theta^* = \arg\min_\theta \mathcal{L}_{\text{task}}(\theta)$ starting from $\theta_0$ instead of random initialization. The Hessian $H = \nabla^2 \mathcal{L}$ at $\theta_0$ has most eigenvalues near zero (the model has learned a low-dimensional manifold), so fine-tuning primarily moves along a small number of important directions. This is why fine-tuning converges in 3-5 epochs while pretraining takes hundreds of passes over the data.

Example:

A pretrained model has never seen radiology reports. Why can fine-tuning on 10,000 reports make it good at radiology?

2 of 4

Catastrophic Forgetting

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{task}} + \lambda \sum_i (\theta_i - \theta_i^{\text{pretrained}})^2$

Fine-tuning moves the weights to lower the task loss, and nothing in that objective protects abilities the task data never exercises. If the shift is large enough, those abilities degrade, which is called catastrophic forgetting. The formula counters it with a penalty: $\mathcal{L}_{\text{task}}$ is the fine-tuning loss, $\theta_i^{\text{pretrained}}$ is weight $i$ before fine-tuning, and $\lambda$ sets how strongly each weight is pulled back. Elastic Weight Consolidation (EWC) refines this by weighting each term by the Fisher information, so important weights move less. Other remedies are a lower learning rate, fewer epochs, mixing general data back in, or training only small adapters such as LoRA.

Mathematical Intuition

Catastrophic forgetting occurs when fine-tuning gradients $\nabla \mathcal{L}_{\text{task}}$ are large in directions that are important for pretrained capabilities. The EWC (Elastic Weight Consolidation) regularizer $\lambda \sum_i F_i(\theta_i - \theta_i^{\text{pre}})^2$ weights the penalty by the Fisher information $F_i = \mathbb{E}[(\partial \log P / \partial \theta_i)^2]$ , which measures how much parameter $\theta_i$ matters for the pretrained distribution. Parameters with high Fisher information are 'important' for pretraining and penalized more strongly during fine-tuning.

Example:

After fine-tuning on legal text, a model can no longer write Python code. What happened?

3 of 4

Feature Extraction vs Fine-Tuning

Feature extraction freezes the pretrained model and uses its hidden states as fixed features for a new, small head, such as a logistic-regression classifier. Only the head is trained, so it is cheap, stable, and hard to overfit, and the base model cannot forget anything. Its limit is that the features were learned for next-token prediction and may not separate the classes the task cares about. Fine-tuning updates some or all of the pretrained weights, so the features themselves adapt. It is more powerful but needs more data, memory, and care. For generation tasks, fine-tuning or in-context learning is needed, because a frozen model cannot change its own outputs.

Mathematical Intuition

Feature extraction uses the pretrained model as a fixed function $f_\theta: \mathcal{X} \to \mathbb{R}^d$ and only trains a head $g_\phi: \mathbb{R}^d \to \mathcal{Y}$ with $|\phi| \ll |\theta|$ parameters. The capacity of the combined model is limited by the representational power of $f_\theta$ — if the pretrained features don't capture task-specific information, no amount of head training will help. Fine-tuning has $|\theta| + |\phi|$ trainable parameters, giving strictly more capacity, but the sample complexity scales as $O(|\theta|/n)$ where $n$ is the number of fine-tuning examples — risking overfitting when $n$ is small.

Example:

For sentiment classification with 100 examples, would you fine-tune or use feature extraction?

4 of 4

Domain Adaptation

Domain adaptation adds a stage between general pretraining and task fine-tuning. The model continues next-token training on unlabelled in-domain text, such as medical papers, legal filings, or a company's code, before it sees any labelled task data. This shifts its vocabulary use, facts, and style toward the domain, so the later task stage has less to learn. Gururangan et al. (2020) found that this domain-adaptive pretraining improved results in all four domains they tested, and that a further pass on the task's own unlabelled text helped again. Use a smaller learning rate than pretraining and mix in some general text, or general skills fade.

Mathematical Intuition

Continued pretraining on domain text reduces the KL divergence $D_{\text{KL}}(P_{\text{domain}} \| P_\theta)$ between the domain's language distribution and the model's learned distribution. Without domain adaptation, the model must generalize from the pretraining distribution $P_{\text{general}}$ — the error is bounded by $\epsilon_{\text{task}} \leq \epsilon_{\text{best}} + d_{\mathcal{H}}(P_{\text{general}}, P_{\text{domain}})$ where $d_{\mathcal{H}}$ is the domain divergence. Continued pretraining directly reduces this divergence term, tightening the error bound before task-specific fine-tuning even begins.

Example:

You have: a general 7B LLM, 50GB of biomedical papers, and 5000 QA pairs for drug interaction queries. What is the optimal fine-tuning pipeline?

Theory Exercise

Problem:

A pretrained LLM achieves 45% on a legal reasoning benchmark. After fine-tuning on 10K legal examples, it reaches 78%. But its general language ability drops from 92% to 71%. How would you maintain both capabilities?

Hints:

Think about multi-task learning
Consider parameter-efficient methods
Think about data mixing strategies

Papers

Howard and Ruder — Universal Language Model Fine-tuning for Text Classification (ULMFiT, ACL 2018)Gururangan et al. — Don't Stop Pretraining: Adapt Language Models to Domains and Tasks (ACL 2020)

Blogs

The State of Transfer Learning in NLP — Sebastian Ruder Fine-tune a Pretrained Model — Hugging Face Transformers docs

Definition

Full fine-tuning continues gradient-based training of every parameter of a pretrained model on a task-specific dataset, starting from the pretrained weights. In supervised fine-tuning (SFT), the dataset contains input-output pairs and the loss is the cross-entropy of the target tokens only. It gives the most capacity of any adaptation method and costs the most memory, since every weight needs gradients and optimizer state.

In this topic

1Full Fine-Tuning Setup
2Supervised Fine-Tuning (SFT)
3Hyperparameter Sensitivity
4When to Use Full Fine-Tuning

1 of 4

Full Fine-Tuning Setup

$\theta^* = \arg\min_\theta \frac{1}{|\mathcal{D}|} \sum_{(x,y) \in \mathcal{D}} \mathcal{L}(f_\theta(x), y)$

The formula is the ordinary empirical-risk objective. $\mathcal{D}$ is the fine-tuning set of input-target pairs $(x, y)$ , $f_\theta$ is the model with parameters $\theta$ , and $\mathcal{L}$ is the loss, usually token cross-entropy. What makes it fine-tuning is the starting point, $\theta$ begins at the pretrained weights. Every parameter gets a gradient, so memory is the bottleneck. In mixed precision with AdamW, each parameter needs about 16 bytes: 2 for BF16 weights, 2 for gradients, 4 for an FP32 master copy, and 8 for Adam's two moments. Typical settings are a learning rate of 1e-5 to 5e-5, 1 to 3 epochs, and warmup then cosine decay.

Mathematical Intuition

Full fine-tuning updates all $N$ parameters with gradient descent: $\theta_{t+1} = \theta_t - \eta \nabla \mathcal{L}_{\text{task}}(\theta_t)$ . Memory requirements are approximately $16N$ bytes in mixed precision: $2N$ (FP16 weights) + $2N$ (FP16 gradients) + $4N$ (FP32 master weights) + $8N$ (FP32 Adam states: $m_t$ and $v_t$ ). For a 7B model: $16 \times 7 \times 10^9 = 112$ GB — fitting on 2 A100-80GB GPUs with activation checkpointing. For 70B: 1.12 TB — requiring 16+ GPUs with model parallelism.

Example:

Full fine-tuning a 7B model with AdamW. How much GPU memory?

2 of 4

Supervised Fine-Tuning (SFT)

SFT trains on (input, target) pairs by concatenating them into one sequence and running ordinary next-token training, with one change: the loss is masked so that only target tokens count. The prompt still passes through the model and conditions every target prediction, but its own tokens produce no gradient. Without the mask, the model would also learn to generate prompts, and long prompts would dominate the loss. With a prompt of $S$ tokens and a response of $R$ tokens, only $R/(S+R)$ of the positions give training signal. SFT is how base models first learn to follow instructions, and it is the first stage of the InstructGPT and Llama 2 Chat pipelines.

Mathematical Intuition

SFT minimizes $\mathcal{L}_{\text{SFT}} = -\sum_{t \in \text{response}} \log P_\theta(y_t \mid x, y_{<t})$ — the loss is computed only on response tokens, not on the prompt. This is crucial: including prompt tokens in the loss would train the model to generate prompts (copying the input style), not to generate good responses. With a prompt of length $S$ and response of length $R$ , the effective training signal density is $R/(S+R)$ — long prompts with short responses waste compute on positions that contribute no gradient.

Example:

SFT example: input = "Summarize: The cat sat on the mat while the dog played outside.", target = "A cat rested on a mat; a dog played outside."

3 of 4

Hyperparameter Sensitivity

Fine-tuning starts from a good solution, so the job is to move a short distance in the right direction. That makes it more sensitive than pretraining. A learning rate that is too high throws away pretrained knowledge and overfits quickly. One that is too low barely changes the model. Small datasets add noise: Mosbach et al. (2020) showed that unstable fine-tuning across random seeds comes mainly from optimisation problems early in training, and that small learning rates with bias correction and more steps fix it. Typical settings are a learning rate of 1e-5 to 5e-5, 1 to 3 epochs, about 5 percent warmup, and weight decay of 0.01 to 0.1.

Mathematical Intuition

Learning rate is the most critical hyperparameter: too high ( $> 5 \times 10^{-5}$ ) causes catastrophic forgetting, too low ( $< 1 \times 10^{-6}$ ) wastes compute. The optimal LR scales approximately as $\eta^* \propto 1/\sqrt{N}$ where $N$ is model size — larger models need smaller updates because each parameter change affects more of the output distribution. Warmup over $w$ steps linearly ramps from 0 to $\eta$ : $\eta_t = \eta \cdot t/w$ . This prevents early gradient spikes when the optimizer's moment estimates $m_t$ and $v_t$ are initialized to zero and not yet calibrated.

Example:

Fine-tuning with LR=1e-4: train loss drops fast but validation loss increases after 100 steps. Diagnosis?

4 of 4

When to Use Full Fine-Tuning

Full fine-tuning pays off when the task needs changes that a small adapter cannot express. That includes large shifts in domain or notation, teaching new skills rather than a new format, very large and high-quality datasets, and continued pretraining. It is also the default when compute is plentiful and quality is what matters. For most applied work, such as style, format, tone, or a narrow task on a few thousand examples, LoRA reaches quality close to full fine-tuning at a fraction of the memory and cost. A further advantage of LoRA is storage: each task adds only megabytes, while full fine-tuning adds another copy of the whole model.

Mathematical Intuition

Full fine-tuning achieves the minimum training loss $\mathcal{L}^*$ because it searches the full parameter space $\mathbb{R}^N$ . LoRA constrains the search to a rank- $r$ subspace of dimension $O(r \cdot d)$ where $d$ is the hidden dimension, potentially missing solutions that require high-rank updates. The gap $\mathcal{L}_{\text{LoRA}} - \mathcal{L}^*$ depends on the intrinsic dimensionality of the fine-tuning task: simple tasks (sentiment analysis) have low intrinsic dimension and LoRA matches full FT, while complex tasks (multilingual translation) may need the full parameter space.

Example:

Fine-tuning a model for a customer support chatbot with 50K conversation examples. Full fine-tuning or LoRA?

Theory Exercise

Problem:

You are fine-tuning a 13B model on 20K instruction examples. After training, the model follows instructions perfectly on the training set but poorly on held-out instructions. What is happening and how do you fix it?

Hints:

Think about overfitting in the context of instruction following
Consider the diversity of your instruction set
Think about regularization strategies

Papers

Mosbach et al. — On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines (ICLR 2021)Ouyang et al. — Training Language Models to Follow Instructions with Human Feedback (InstructGPT, 2022)

Blogs

Fine-tune a Pretrained Model — Hugging Face Transformers docs SFT Trainer — Hugging Face TRL docs

Definition

LoRA (low-rank adaptation) freezes a pretrained weight matrix $W$ and learns its update as a product $BA$ of two thin matrices with inner rank $r$ , much smaller than the matrix width. Only $A$ and $B$ are trained. QLoRA applies LoRA on top of a base model quantized to 4-bit NF4, with gradients flowing through the frozen quantized weights into the adapters.

In this topic

1LoRA: Low-Rank Adaptation
2Why Low-Rank Works
3QLoRA: Quantized LoRA
4LoRA Hyperparameters

1 of 4

LoRA: Low-Rank Adaptation

$W' = W + \Delta W = W + BA, \quad B \in \mathbb{R}^{d \times r}, A \in \mathbb{R}^{r \times d}, \quad r \ll d$

LoRA keeps the pretrained matrix $W$ frozen and learns the update $\Delta W = BA$ , as in the formula. $B$ is $d \times r$ , $A$ is $r \times d$ , and the rank $r$ is typically 8 to 64 while $d$ is 4,096 or more. The forward pass computes $Wx + BAx$ . $A$ starts as small random values and $B$ starts at zero, so the model begins exactly at the pretrained function. Only $2rd$ parameters per matrix are trained, and the optimizer state shrinks with them. After training, $BA$ can be merged into $W$ , so inference has no extra latency. The limit is that a rank- $r$ update cannot express changes that need high rank.

Mathematical Intuition

LoRA decomposes the weight update as $\Delta W = BA$ where $B \in \mathbb{R}^{d \times r}$ , $A \in \mathbb{R}^{r \times d}$ , and $r \ll d$ . The forward pass becomes $h = (W + BA)x = Wx + BAx$ . Trainable parameters per layer: $2rd$ versus $d^2$ for full fine-tuning. For $d = 4096$ and $r = 16$ : $2 \times 16 \times 4096 = 131{,}072$ vs $16{,}777{,}216$ — a 128 $\times$ reduction. Applied to all attention matrices ( $W_Q, W_K, W_V, W_O$ ) across $L$ layers: $4 \times 2rd \times L$ total trainable parameters.

Example:

A weight matrix $W$ has shape $4096 \times 4096$ (16.8M params). LoRA with $r = 16$ . How many trainable params?

2 of 4

Why Low-Rank Works

A weight update is low rank if a few directions explain almost all of it. Aghajanyan et al. (2020) measured the intrinsic dimension of fine-tuning and found it tiny. Optimising only 200 parameters, randomly projected into RoBERTa's full weight space, reached 90 percent of full fine-tuning performance on MRPC, and larger pretrained models had lower intrinsic dimension. Hu et al. (2021) found that LoRA updates of rank 1 to 8 were often as good as larger ranks for GPT-3. The interpretation is that pretraining already built the needed features, and adaptation mostly re-weights a few of them. Tasks that teach new knowledge, such as a new language, need higher rank.

Mathematical Intuition

The intrinsic dimensionality hypothesis states that fine-tuning updates $\Delta W$ lie in a low-dimensional subspace. Empirically, the singular values of $\Delta W$ decay rapidly: the top $r$ singular values capture 90%+ of the spectral energy for $r \ll d$ . Specifically, if $\sigma_1 \geq \sigma_2 \geq \ldots$ are the singular values of $\Delta W$ , then $\sum_{i=1}^r \sigma_i^2 / \sum_{i=1}^d \sigma_i^2 > 0.9$ for $r \approx 8-64$ even when $d = 4096$ . This means the true weight change during fine-tuning is nearly rank-deficient — LoRA directly exploits this structure.

Example:

If you SVD the weight change $\Delta W$ after full fine-tuning, what do you expect to see?

3 of 4

QLoRA: Quantized LoRA

$\text{Memory} \approx \frac{N}{2} \text{ (4-bit base)} + r \cdot d \cdot L \cdot 4 \text{ (BF16 LoRA)} + \text{optimizer states}$

QLoRA shrinks the frozen base model so it fits on one GPU. The base weights are stored in NF4, a 4-bit format whose 16 levels are placed at quantiles of a normal distribution, which suits neural-network weights. Each block of 64 weights shares a scale, and double quantization compresses those scales too, from 0.5 to about 0.127 bits per parameter. LoRA adapters stay in BF16 and are the only trained weights. The formula reads: about $N/2$ bytes for the 4-bit base, plus the adapters, plus optimizer state for the adapters only. Paged optimizers spill memory spikes to CPU RAM. Dettmers et al. matched 16-bit fine-tuning quality this way.

Mathematical Intuition

QLoRA combines 4-bit quantization with LoRA: the base model $W$ is stored in NF4 (4-bit NormalFloat) format at 0.5 bytes/parameter, while LoRA adapters $A, B$ remain in BF16 at 2 bytes/parameter. Memory: $0.5N + 4rLd$ bytes versus $16N$ for full FP16 fine-tuning. For 65B parameters with $r = 16$ , $L = 80$ , $d = 8192$ : $32.5$ GB (base) + $0.08$ GB (adapters) $\approx 33$ GB — fitting on a single A100-40GB. The NF4 format is information-theoretically optimal for normally distributed weights: it minimizes quantization error $\mathbb{E}[(W - Q(W))^2]$ under the assumption $W \sim \mathcal{N}(0, \sigma^2)$ .

Example:

Fine-tuning Llama 2 70B with QLoRA (r=16). Estimate GPU memory.

4 of 4

LoRA Hyperparameters

Four settings matter most. The rank $r$ sets capacity, usually 8 to 64. The alpha $\alpha$ scales the update, so the layer computes $Wx + (\alpha/r)BAx$ . Holding $\alpha/r$ fixed keeps the update's size similar when you change rank, which means less learning-rate retuning. Target modules decide which matrices get adapters. The original paper used only Q and V, but Raschka's and QLoRA's experiments found that adapting all linear layers, including the MLP, usually helps more than raising rank. LoRA dropout of 0.05 to 0.1 regularises small datasets. The learning rate is higher than in full fine-tuning, often 1e-4 to 2e-4.

Mathematical Intuition

Rank $r$ controls the expressiveness vs efficiency trade-off: total trainable parameters scale as $O(r)$ , but task performance saturates quickly. Empirically, $r = 8$ matches $r = 256$ within 1% accuracy on most benchmarks for 7B models. The scaling factor $\alpha/r$ normalizes the adapter output magnitude: $\Delta W x = (\alpha/r) \cdot BA \cdot x$ . Setting $\alpha = 2r$ (the common default) means doubling $r$ doubles the adapter capacity without changing output scale. Target modules also matter: applying LoRA to $W_Q, W_V$ only (not $W_K, W_O$ ) captures 95% of the quality at 50% of the adapter parameters.

Example:

LoRA with $r=16$ , $\alpha=32$ . What is the effective scaling? What if $r=32$ , $\alpha=32$ ?

Theory Exercise

Problem:

You need to fine-tune a 70B model for two different tasks: legal document analysis and medical report generation. You want to switch between tasks at inference time. How would you design this with LoRA?

Hints:

Think about LoRA as modular adapters
Consider that multiple LoRA adapters can be trained on the same base model
Think about hot-swapping adapters at inference time

Papers

Hu et al. — LoRA: Low-Rank Adaptation of Large Language Models (2021)Dettmers et al. — QLoRA: Efficient Finetuning of Quantized LLMs (2023)

Blogs

Making LLMs Even More Accessible with bitsandbytes, 4-bit Quantization and QLoRA — Hugging Face Practical Tips for Finetuning LLMs Using LoRA — Sebastian Raschka

Definition

Instruction tuning is supervised fine-tuning on (instruction, response) pairs drawn from many tasks, each phrased as a natural-language request, so that a pretrained model learns to produce the requested output instead of continuing the text. The loss is computed on response tokens only. Its aim is generalisation to new instructions, not skill at the specific training tasks.

In this topic

1What is Instruction Tuning
2Instruction Dataset Design
3Multi-Task Instruction Tuning
4Chat Templates and Formatting

1 of 4

What is Instruction Tuning

Instruction tuning is SFT on examples such as "Summarize this article", "Translate to French", or "Write a Python function that reverses a list", each paired with a good response. The loss covers only the response, so the model learns to map a request to an answer. The striking result is generalisation. FLAN (Wei et al., 2021) tuned a 137B model on over 60 tasks phrased as instructions, and it beat zero-shot GPT-3 175B on 20 of 25 unseen tasks. The model had not learned those tasks. It had learned the meta-skill of reading an instruction and doing what it says. The knowledge itself came from pretraining.

Mathematical Intuition

Instruction tuning transforms $P(\text{next token} \mid \text{context})$ into $P(\text{helpful response} \mid \text{instruction})$ . The key distribution shift: pretraining maximizes likelihood over web text (including toxic, unhelpful, and irrelevant content), while instruction tuning maximizes likelihood over curated (instruction, response) pairs that exemplify desired behavior. Even with just 1,000-10,000 high-quality examples (less than 0.001% of pretraining data), instruction tuning dramatically changes the model's output distribution — evidence that the behavioral knowledge was already latent in the pretrained weights.

Example:

A base model and an instruction-tuned model both receive: 'List 3 benefits of exercise.' How do they respond?

2 of 4

Instruction Dataset Design

A good instruction dataset covers many task types, including question answering, summarisation, extraction, rewriting, coding, math, reasoning, and open-ended writing. It phrases requests in many ways, formal and casual, terse and detailed, with and without examples. Every response must be correct and complete, and it should use the formatting a user would want. Sources vary: FLAN converts academic datasets with templates, Alpaca generated 52K examples from text-davinci-003 using the Self-Instruct method, and OpenAssistant and Dolly were written by people. Templated data is cheap but repetitive. LIMA (Zhou et al., 2023) showed that 1,000 carefully chosen examples can produce a strong assistant from a 65B model.

Mathematical Intuition

Dataset diversity matters more than size: FLAN-v2 showed that mixing $T$ distinct task types improves zero-shot performance on held-out tasks as $\text{acc} \propto \log(T)$ — logarithmic scaling with task diversity. Each task type teaches a distinct input-output mapping (summarization, QA, translation, code generation). With $T$ tasks and $n_i$ examples per task, the total dataset size is $\sum_i n_i$ , but the effective training signal depends on $T$ more than $\sum_i n_i$ . This is why 10 tasks with 1,000 examples each outperforms 1 task with 100,000 examples.

Example:

Compare 100K instructions from a template vs 10K human-written instructions. Which produces a better model?

3 of 4

Multi-Task Instruction Tuning

$\text{Generalization} \propto \text{number of task types}^{\alpha}, \quad \alpha \approx 0.1-0.3$

Mixing many distinct tasks is what makes instruction tuning generalise. The formula is a rough empirical trend, not a law: held-out performance grows with the number of task types raised to a small power $\alpha$ , so each new kind of task helps more than more examples of an old one. Chung et al. (2022) scaled to 1,836 tasks and found that Flan-PaLM 540B beat PaLM 540B by 9.4 points on average, with most of the gain arriving by a few hundred tasks. Big tasks must not swamp small ones, so mixtures cap examples per task or sample tasks by a tempered share of their size.

Mathematical Intuition

Multi-task learning balances $\mathcal{L}_{\text{total}} = \sum_i w_i \mathcal{L}_i$ where $w_i$ is the weight for task $i$ . The optimal weights depend on task difficulty and dataset size: setting $w_i \propto \sqrt{D_i}$ (proportional to square root of dataset size) prevents large datasets from dominating. Temperature-based sampling draws from task $i$ with probability $p_i \propto D_i^{1/T}$ where temperature $T$ controls uniformity: $T = 1$ samples proportionally to dataset size, $T \to \infty$ samples uniformly across tasks. The FLAN recipe uses $T \approx 2$ as a compromise.

Example:

Model A trained on 100K examples from 5 task types. Model B trained on 50K examples from 50 task types. Which generalizes better to a new task?

4 of 4

Chat Templates and Formatting

A chat model sees a conversation as one token sequence, with special tokens marking the system prompt, each user turn, and each assistant turn. Llama 2 Chat uses [INST], [/INST], and <<SYS>> markers, while ChatML models wrap each turn in im_start and im_end tokens. During SFT the model learns what each marker means: after the end of a user turn, write an assistant answer, and after the answer, emit the end token. The template is part of the model's interface, and the tokenizer usually ships it as a Jinja template. Missing or wrong markers put the input outside the fine-tuning distribution, and quality drops sharply.

Mathematical Intuition

Chat templates structure the token sequence with special tokens that the model learns to associate with behavioral modes: $[\text{SYSTEM}] \oplus s \oplus [\text{USER}] \oplus u \oplus [\text{ASSISTANT}]$ . The model's conditional distribution $P(y \mid [\text{ASSISTANT}], u, s)$ is sharply different from $P(y \mid u)$ because the special tokens activate instruction-following behaviors. During training, the loss mask ensures gradients come only from assistant tokens: $\mathcal{L} = -\sum_{t \in \text{assistant}} \log P(y_t \mid y_{<t})$ , not from system or user tokens.

Example:

Why does using the wrong chat template produce poor results even with a well-fine-tuned model?

Theory Exercise

Problem:

You are building an instruction dataset for fine-tuning a code-focused LLM. Design the dataset composition (task types, difficulty levels, languages). How many examples do you need?

Hints:

Think about the diversity of coding tasks
Consider different difficulty levels
Think about what makes code instruction data high-quality

Papers

Wei et al. — Finetuned Language Models Are Zero-Shot Learners (FLAN, 2021)Chung et al. — Scaling Instruction-Finetuned Language Models (Flan-PaLM and Flan-T5, 2022)

Blogs

Chat Templates — Hugging Face Transformers docs Alpaca: A Strong, Replicable Instruction-Following Model — Stanford CRFM

Definition

Data preparation for fine-tuning is the process of collecting, filtering, formatting, deduplicating, decontaminating, and balancing training examples so that the dataset matches the target behaviour and deployment distribution. It covers quality filtering, template validation, removal of benchmark overlaps, and controlled augmentation. Its output is a dataset in which every example demonstrates exactly the behaviour the model should learn.

In this topic

1Data Quality Over Quantity
2Formatting and Consistency
3Decontamination
4Data Augmentation

1 of 4

Data Quality Over Quantity

In fine-tuning, the model copies whatever its examples do, so quality defines the target. LIMA (Zhou et al., 2023) fine-tuned a 65B LLaMA on only 1,000 carefully chosen prompts and responses, with no RLHF. People judged its answers equivalent to or better than GPT-4's in 43 percent of comparisons. The authors concluded that almost all knowledge comes from pretraining, and fine-tuning mainly teaches format and style. Quality has several parts: correct facts, complete answers, consistent style, diverse tasks, and the right format. Large noisy sets are still useful once filtered, for example by scoring each example with a classifier or a strong LLM judge and keeping the top slice.

Mathematical Intuition

The LIMA paper demonstrated that 1,000 carefully curated examples outperform 50,000 noisy examples. The information-theoretic explanation: each high-quality example provides $I_{\text{high}}$ bits of task-relevant information versus $I_{\text{low}}$ for noisy examples where $I_{\text{high}} \gg I_{\text{low}}$ . The effective dataset size is $\sum_i I_i$ , not the count. A single example with a factual error can establish an incorrect pattern that persists for thousands of correct examples to undo — the 'one bad apple' effect where error correction requires $O(1/\eta)$ opposing examples where $\eta$ is the learning rate.

Example:

You have 100K automatically generated instruction-response pairs and 5K human-verified ones. How do you use both?

2 of 4

Formatting and Consistency

A fine-tuned model learns style as readily as content. If some responses end with a period and others do not, or some use markdown lists and others plain prose for the same kind of request, the model learns both habits and picks one at random at inference. Template errors are worse: a missing end-of-turn token teaches the model never to stop, and a stray system prompt teaches it to expect one. Consistency means one chat template, one style guide for responses, and automated checks before training. Check for non-empty responses, balanced special tokens, sequence lengths under the context limit, and correct masking of the prompt tokens.

Mathematical Intuition

Inconsistent formatting adds noise to the gradient signal: if the same task appears with two formats, the model's gradient at each step partly optimizes for format and partly for content. The effective learning rate for content-specific features drops by a factor of $(1 - v)$ where $v$ is the fraction of gradient variance attributable to formatting inconsistency. With $k$ different formats, $v \approx (k-1)/k$ in the worst case. Standardizing to a single chat template eliminates this variance, concentrating all gradient signal on learning the actual task.

Example:

Some examples end with a period, others don't. Some use markdown formatting, others don't. Does this matter?

3 of 4

Decontamination

A benchmark measures how well a model generalises to unseen questions. If benchmark items leak into training, the model can recall answers instead of solving problems, and the score stops meaning anything. Leaks are common because fine-tuning data is scraped, generated, or copied from sources that quote benchmarks. Decontamination removes training examples that overlap evaluation sets. The usual test is n-gram overlap, where GPT-3 flagged 13-gram matches, plus near-duplicate detection with MinHash for paraphrased copies. Check every benchmark you will report, such as MMLU, GSM8K, and HumanEval. Report the method used, and when possible keep a private held-out set that never leaves your team.

Mathematical Intuition

Decontamination prevents benchmark inflation where the model memorizes test set answers during training. The detection method computes n-gram overlap: for each test example $x_{\text{test}}$ , check if any training example contains a contiguous $n$ -gram match of length $\geq k$ (typically $k = 13$ tokens). The probability of a random 13-gram match in a vocabulary of $V = 50{,}000$ tokens is $V^{-13} \approx 10^{-61}$ — essentially zero, so any match indicates true contamination. Even partial contamination (similar but not identical examples) can inflate scores by 2-5 percentage points.

Example:

Your fine-tuning data contains 50 HumanEval problems with solutions. What happens?

4 of 4

Data Augmentation

Augmentation expands a small seed set when collecting more human data is too slow or costly. Common methods rephrase instructions, vary inputs such as the numbers in a math problem, add harder or edge-case versions, translate into other languages, and have a strong LLM write new examples. Self-Instruct (Wang et al., 2022) bootstrapped 52K instructions from 175 seed tasks using GPT-3's own generations, filtered for quality and similarity, and improved GPT-3 by 33 points on Super-NaturalInstructions. The risks are errors copied from the generator, low diversity, and license limits on generated data. Verify outputs by running code, checking math, or human review on a sample.

Mathematical Intuition

Self-instruct augmentation uses the model itself to generate new (instruction, response) pairs, expanding a seed dataset of $n_0$ examples to $n_0 + kn_0 = n_0(1 + k)$ examples. The diversity of augmented data is bounded by the model's generalization ability: $\text{diversity}(\text{augmented}) \leq \text{diversity}(\text{model})$ , which itself is bounded by the pretraining data. Back-translation augments by paraphrasing: instruction $\to$ response $\to$ rephrase instruction $\to$ verify response still applies. Each augmented example adds diminishing information: the $m$ -th augmentation of the same seed contributes $\approx I_1 / m$ bits.

Example:

Original instruction: "Write a function to sort a list." How would you augment this?

Theory Exercise

Problem:

You are preparing a fine-tuning dataset for a multilingual customer support bot. The data is 80% English, 15% Spanish, and 5% Japanese. Users will query 50% English, 30% Spanish, 20% Japanese. What adjustments would you make?

Hints:

Compare the training distribution to the deployment distribution
Think about upsampling underrepresented languages
Consider the impact on per-language quality

Papers

Zhou et al. — LIMA: Less Is More for Alignment (2023)Wang et al. — Self-Instruct: Aligning Language Models with Self-Generated Instructions (ACL 2023)

Blogs

Large-scale Near-deduplication Behind BigCode — Hugging Face Datasets documentation — Hugging Face

Transfer Learning

Full Fine-tuning

LoRA & QLoRA

Instruction Tuning

Data Preparation

5 topics in this chapter

Transfer Learning

Definition

In this topic

Why Transfer Learning Works for LLMs

Catastrophic Forgetting

Feature Extraction vs Fine-Tuning

Domain Adaptation

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

Full Fine-tuning

Definition

In this topic

Full Fine-Tuning Setup

Supervised Fine-Tuning (SFT)

Hyperparameter Sensitivity

When to Use Full Fine-Tuning

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

LoRA & QLoRA

Definition

In this topic

LoRA: Low-Rank Adaptation

Why Low-Rank Works

QLoRA: Quantized LoRA

LoRA Hyperparameters

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

Instruction Tuning

Definition

In this topic

What is Instruction Tuning

Instruction Dataset Design

Multi-Task Instruction Tuning

Chat Templates and Formatting

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

Data Preparation

Definition

In this topic

Data Quality Over Quantity

Formatting and Consistency

Decontamination

Data Augmentation

Theory Exercise

Problem:

Hints: