Chapter 6: RLHF & Alignment
Learn how to align LLMs with human values and preferences. Master Reinforcement Learning from Human Feedback (RLHF)---the technique that transformed GPT-3 into ChatGPT---understand reward modeling, policy optimization with PPO, the simpler Direct Preference Optimization (DPO) alternative, Constitutional AI for scalable oversight, and the challenges of collecting human preference data.
Chapter Overview
Instruction tuning teaches a model to follow instructions, but it does not teach it to be helpful, harmless, or honest. A model might follow the instruction "explain how to hack a website" perfectly---and that is exactly the problem. Alignment ensures models behave in ways that are beneficial and safe.
Reinforcement Learning from Human Feedback (RLHF) is the technique that made ChatGPT possible. The key insight: instead of training on (input, correct output) pairs, we train on (output A is better than output B) preferences. Humans rank model outputs, a reward model learns to predict these preferences, and the LLM is optimized to maximize the reward model's score.
However, RLHF is complex and unstable. Recent alternatives like Direct Preference Optimization (DPO) achieve similar results with a simpler training pipeline. Constitutional AI (CAI) reduces the need for human annotation by having the model critique and revise its own outputs.
This chapter covers:
- Reward Modeling: Training a model to predict human preferences
- PPO for LLMs: The RL algorithm that optimizes the policy against the reward model
- DPO: A simpler alternative that eliminates the need for a separate reward model
- Constitutional AI: Scalable alignment through AI-generated feedback
- Human Preference Data: How to collect, curate, and validate preference annotations
Chapter Roadmap
Click any topic to jump in
Reward Modeling
Training a model to predict human preferences — the bridge between human judgment and optimization.
Preference Data
Collecting comparison data — how to gather and quality-control human preference annotations.
RL-based vs direct optimization
PPO for LLMs
Proximal Policy Optimization — the RL algorithm that fine-tunes LLMs against the reward model.
DPO
Direct Preference Optimization — skipping the reward model by optimizing preferences directly.
Constitutional AI
Self-improvement through AI-generated critiques — reducing reliance on human feedback.
Sign up to unlock this chapter
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.