Chapter 6: RLHF & Alignment

The formula asks for a policy $\pi$ that earns high reward $r(x, y)$ on prompts $x$ from the dataset $\mathcal{D}$, while staying close to the reference policy $\pi_{\text{ref}}$, the frozen SFT model. Closeness is the KL divergence $D_{KL}$, weighted by $\beta$. The penalty matters because the reward model is only accurate near the responses it was trained on. Far from them its scores are unreliable and easy to exploit. In practice the KL term is estimated per token from log-probability differences and subtracted from the reward. Bai et al. (2022) found reward grew roughly linearly with the square root of KL. Typical $\beta$ values are 0.01 to 0.2.

PPO updates the policy with the objective in the formula. The ratio $\pi(y|x)/\pi_{\text{old}}(y|x)$ compares the current policy with the one that generated the samples, and $A(x, y)$ is the advantage, how much better the response did than expected. Multiplying them gives the standard policy-gradient surrogate. Clipping the ratio to $[1-\epsilon, 1+\epsilon]$, with $\epsilon$ usually 0.2, and taking the minimum of the clipped and unclipped terms removes any incentive to move a probability far in one update. That lets PPO reuse each expensive batch of generated responses for several gradient steps. For LLMs the ratio and advantage are computed per token.

The policy gradient multiplies each response's log-probability gradient by a score. If that score is the raw reward, every response with a positive reward is reinforced, good or bad, and learning is slow and noisy. Subtracting a baseline fixes this. The value function $V(x)$ in the formula estimates the reward the current policy expects on prompt $x$, and the advantage $A = r - V(x)$ asks whether this response beat that expectation. The gradient stays unbiased, but its variance drops sharply. In RLHF the critic is usually another copy of the model with a scalar head, often started from the reward model and trained alongside the policy.

PPO-based RLHF is the hardest stage of the LLM pipeline to run. It needs four models: the policy and value model, both trained, plus the frozen reference and reward models. Each step alternates slow autoregressive generation with scoring and several PPO epochs, so the GPUs switch between inference and training. Many details matter, including reward normalisation, KL estimation, advantage whitening, the learning rate, $\beta$, $\epsilon$, and batch size. Huang et al. listed over 20 of them when reproducing OpenAI's summarisation results. Watch for rising KL, collapsing output diversity, and growing response length, the usual signs of instability or reward hacking.

The KL-regularised RLHF objective has a known optimal policy: $\pi^*(y|x)$ is proportional to $\pi_{\text{ref}}(y|x)\,e^{r(x,y)/\beta}$. Solving for the reward gives $r = \beta \log(\pi^*/\pi_{\text{ref}})$ plus a term that depends only on $x$, which cancels in a Bradley-Terry comparison. Substituting this implicit reward into the reward-model loss gives the DPO loss in the formula. $\pi_\theta$ is the policy being trained, $\pi_{\text{ref}}$ the frozen SFT model, and $y_w$ and $y_l$ the chosen and rejected responses. Each log ratio says how much more likely the policy makes a response than the reference does. The loss pushes the chosen ratio above the rejected one.

DPO trains like SFT. It makes one pass over a fixed dataset of pairs with a standard loss, needs only the policy and a frozen reference, and has one main hyperparameter, $\beta$. The reference log-probabilities can even be computed once in advance. It is stable and cheap, which is why open models such as Zephyr and Tulu 2 used it. The costs come from being offline. The policy never tries new responses and gets feedback on them, so it learns only from the pairs it is given. An explicit reward model can also score fresh samples, generalise across prompts, and be reused. Online and iterative DPO variants narrow this gap.

In DPO, $\beta$ plays the role of the KL coefficient in RLHF. It scales the log-ratio margin inside the sigmoid. With a large $\beta$, a small change in the log ratios already saturates the sigmoid, so the loss is satisfied early and the policy stays close to the reference. With a small $\beta$, the policy must move much further from the reference to reach the same loss, so it follows the preferences more aggressively. That improves alignment on the data but raises the risk of overfitting and degenerate text. The DPO paper used values from 0.1 to 0.5, and 0.1 is the common default. Always tune it on held-out preferences.

Each variant fixes one weakness of DPO. IPO (Azar et al., 2023) points out that with deterministic preferences DPO can push the margin toward infinity and overfit. It replaces the log-sigmoid with a squared loss that pulls the margin toward a fixed target, $1/(2\tau)$. KTO (Ethayarajh et al., 2024) needs no pairs, only responses labelled desirable or undesirable, and uses a prospect-theory value function that weighs losses more than gains. It matched pairwise methods from 1B to 30B parameters. ORPO (Hong et al., 2024) adds an odds-ratio penalty to the SFT loss, so one stage does both steps without a reference model.

CAI has two phases. In the supervised phase, a helpful but unsafe model answers harmful prompts. It is then asked to critique its answer against a randomly chosen principle and to rewrite it, sometimes over several rounds. The original model is fine-tuned on the revised answers. In the RL phase, the fine-tuned model produces pairs of answers, a model picks the better one according to a principle, and those AI preferences train a preference model. RL against that model is called RLAIF. Human helpfulness labels were still used. The result was a model that was both more harmless and less evasive, because it explained its objections instead of refusing silently.

A human preference label covers one comparison. A principle such as "choose the response that is less likely to help someone cause harm" can be applied to any number of comparisons, at almost no cost per example. The constitution also makes the target explicit: the rules can be read, argued about, and changed, and changing them means rerunning the pipeline, not relabelling everything. Principles have limits. They can conflict, as when helpfulness and caution pull in different directions. They depend on the model interpreting them correctly, and vague wording produces inconsistent labels. Anthropic publishes the constitution it uses for Claude.

In RLAIF, an off-the-shelf LLM labels preference pairs in place of people, usually by being shown the prompt, two responses, and a principle or rubric, and asked which is better. Asking it to reason first, and averaging over both orders to remove position bias, improves the labels. The labels then train a reward model as usual. Lee et al. (2023) found RLAIF comparable to RLHF on summarisation and helpful and harmless dialogue, even when the labeller was the same size as the policy. LLM judges also agree with human preferences over 80 percent of the time in MT-Bench tests. They share model biases such as favouring verbose or self-similar text.

Red teaming is adversarial testing: people or models deliberately try to make the system produce harmful output before users do. Targets include dangerous instructions, harassment, bias, privacy leaks, misinformation, and prompt injection. Ganguli et al. (2022) released 38,961 human red-team attacks and found that RLHF models became harder to attack as they grew. Perez et al. (2022) used one language model to generate test cases against another, uncovering tens of thousands of offensive replies in a 280B chatbot. Findings feed back into training data, the constitution, and filters. Because every fix invites new attacks, red teaming is continuous, not a one-time audit.

Good preference data starts with the labelling task, not the model. Write guidelines that define what better means for each dimension, such as helpfulness, honesty, and harmlessness, and say how to trade them off, with worked examples of hard cases. Run calibration rounds where labellers compare answers and discuss disagreements. Mix in gold items with known answers to track each labeller's accuracy. Collect several labels per item for a sample, so agreement can be measured. Let labellers mark ties and say how strong a preference is. Llama 2 recorded strength on a four-level scale, from significantly better to negligibly better, and used it as a margin.

Labellers are fast, tired, and human, and their shortcuts become model behaviour. Length bias favours longer answers, since they look more thorough. Position bias favours whichever response is shown first, or second. Confidence bias favours assured answers over hedged ones, even when the hedge is correct. Sycophancy bias favours answers that agree with the user's stated view. Sharma et al. (2023) found that both humans and preference models sometimes preferred convincing sycophantic answers over correct ones, and that optimising against such models can trade truthfulness for agreement. Countermeasures include random order, length-matched pairs, fact-checking tasks, and prompts built so the correct answer disagrees with the user.

Preference datasets range from tens of thousands to millions of comparisons. InstructGPT's reward model was trained on rankings of 4 to 9 responses for each of tens of thousands of prompts. Anthropic's helpful and harmless data had over 160,000 comparisons, and Llama 2 used over 1 million new comparisons, collected in weekly batches as the model improved. Cost per label varies widely. Crowd workers cost about a dollar or two per comparison, specialists much more, and domain experts in medicine or law the most. Costs are cut by using AI labels for easy cases, reserving experts for hard ones, and collecting rankings instead of single choices.

The format fixes how much information each judgment carries. A binary choice between two responses is fastest and most reliable, and it gives one pair. A ranking of $k$ responses gives $\binom{k}{2}$ pairs, though they are correlated because they come from one ordering. A Likert rating, such as 1 to 5, captures how good each response is, but people calibrate scales differently, so ratings are noisier across labellers. Per-aspect labels, such as helpfulness and safety judged separately, let you train separate reward models, as Llama 2 did, or weight the aspects later. Single good or bad labels suit KTO. Choose the format to match the algorithm and your labelling budget.