Chapter 1: Introduction to Reinforcement Learning
Build the mental model behind reinforcement learning: an agent that learns by interacting with an environment, guided only by reward. Understand how RL differs from supervised learning, formalize the agent-environment loop, and meet the core ideas — rewards, returns, policies, value functions, and the exploration-exploitation tradeoff.
Chapter Overview
Reinforcement learning (RL) is the study of how an agent should act in an environment to maximize a cumulative reward signal. Unlike supervised learning, there is no labeled dataset telling the agent the correct action — instead the agent must discover which actions are good by trying them and observing the consequences.
This trial-and-error loop, combined with the idea that actions can have delayed effects, is what makes RL both powerful and challenging. A move in a game, a trade in a market, or a torque applied to a robot joint may only reveal whether it was wise many steps later.
This chapter establishes the vocabulary you will use for the rest of the plan. You will learn to describe any sequential decision problem in terms of states, actions, and rewards; understand why we discount future reward; and see how policies and value functions let an agent reason about long-term consequences. We close with the exploration-exploitation dilemma — the fundamental tension at the heart of every RL algorithm.
This chapter covers:
- What is RL: Learning from interaction versus learning from labels
- The Agent-Environment Interface: States, actions, rewards, and the interaction loop
- Rewards & Returns: The reward hypothesis and discounted return
- Policies & Value Functions: Mapping states to actions and estimating long-term value
- Exploration vs Exploitation: Why an agent must sometimes act sub-optimally to learn
Chapter Roadmap
Click any topic to jump in
What is RL
Learning by interaction: an agent maximizes cumulative reward with no labeled answers, only feedback from the environment.
Agent-Environment Loop
At each step the agent observes a state, picks an action, and receives a reward and next state — the canonical RL interaction.
Policies choose actions; value functions score how much return a state or action is worth.
Rewards & Returns
Goals are encoded as a scalar reward. The return is the discounted sum of future rewards the agent actually tries to maximize.
Policies & Values
A policy maps states to actions; value functions estimate the expected return, letting the agent reason about the long term.
Exploration vs Exploitation
To find the best actions the agent must explore unknown options, while still exploiting what it already knows works.
Reinforcement learning sits between supervised and unsupervised learning, but the difference is not a matter of degree — it is a different kind of feedback entirely. In supervised learning a teacher supplies the correct label for every input, so the learner is told what it should have done. In unsupervised learning there are no labels at all, only structure to discover. Reinforcement learning has feedback, but that feedback is evaluative rather than instructive: a reward says how good the chosen action was without ever revealing which action would have been best. Two further complications follow from that. The data is not a fixed dataset handed to you in advance — it is generated by the agent's own choices, so an agent that never tries an action never learns anything about it. And the reward may arrive many steps after the decision that earned it, leaving the agent to work out which of its past actions deserve the credit. Those three properties — evaluative feedback, self-generated data, and delayed consequences — are what make reinforcement learning its own field rather than a variant of supervised learning.
Definition
Reinforcement learning is the study of how an agent should choose actions in an environment so as to maximize cumulative reward over time. The agent is never told the correct action; it receives only a scalar reward signal, and must discover a good policy by trying actions and observing their consequences.
In this topic
Learning From Interaction
An RL agent learns a behaviour by repeatedly interacting with an environment. At each step it takes an action, the environment responds with a new situation and a numerical reward, and the agent revises its strategy to earn more reward over time. The crucial structural difference from supervised learning is that there is no fixed training set: the data the agent learns from is generated by its own choices. That creates a feedback loop with no analogue in supervised learning: a poor policy visits poor states, collects uninformative data, and stays poor. Good exploration is not a refinement here, it is a precondition for learning anything at all.
RL optimizes an objective over trajectories: maximize , the expected total reward, where the expectation is taken over the agent's own action choices and the environment's dynamics.
Why can't we treat learning to play a game as a standard supervised-learning problem where each board position is labeled with the 'correct' move?
Evaluative vs Instructive Feedback
Instructive feedback names the correct action, so a single example fully determines what the learner should do at that input. Evaluative feedback only scores the action that was actually taken, which means a reward of is uninformative in isolation — you cannot tell whether an unchosen alternative would have scored or . The only way to find out is to try it, and trying it costs whatever the alternative was worth. This is precisely why exploration is a structural requirement of reinforcement learning rather than a heuristic: without it, the value of every untried action stays permanently unknown.
If is the best action, supervised learning observes directly, while RL only observes for the chosen and must infer from samples.