Loading...
Learn how to align LLMs with human values and preferences. Master Reinforcement Learning from Human Feedback (RLHF)---the technique that transformed GPT-3 into ChatGPT---understand reward modeling, policy optimization with PPO, the simpler Direct Preference Optimization (DPO) alternative, Constitutional AI for scalable oversight, and the challenges of collecting human preference data.
Instruction tuning teaches a model to follow instructions, but it does not teach it to be helpful, harmless, or honest. A model might follow the instruction "explain how to hack a website" perfectly---and that is exactly the problem. Alignment ensures models behave in ways that are beneficial and safe.
Reinforcement Learning from Human Feedback (RLHF) is the technique that made ChatGPT possible. The key insight: instead of training on (input, correct output) pairs, we train on (output A is better than output B) preferences. Humans rank model outputs, a reward model learns to predict these preferences, and the LLM is optimized to maximize the reward model's score.
However, RLHF is complex and unstable. Recent alternatives like Direct Preference Optimization (DPO) achieve similar results with a simpler training pipeline. Constitutional AI (CAI) reduces the need for human annotation by having the model critique and revise its own outputs.
This chapter covers:
Click any topic to jump in
Training a model to predict human preferences — the bridge between human judgment and optimization.
Collecting comparison data — how to gather and quality-control human preference annotations.
RL-based vs direct optimization
Proximal Policy Optimization — the RL algorithm that fine-tunes LLMs against the reward model.
Direct Preference Optimization — skipping the reward model by optimizing preferences directly.
Self-improvement through AI-generated critiques — reducing reliance on human feedback.
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.