What Is Reinforcement Learning? How AI Learns From Rewards

What is reinforcement learning?

In reinforcement learning, a learner called the agent interacts with a setting called the environment. It acts over a series of steps, and after each action it receives a numeric reward that scores the result.

The goal is not to maximise the next reward but the total reward over time, which forces the agent to weigh consequences rather than chase the nearest payoff.

The standard reference is Sutton and Barto's Reinforcement Learning: An Introduction, now in its second edition from MIT Press. Almost every RL tutorial uses its framing of agent, environment, reward and policy, and so does this one.

A useful mental model is training a dog, because you cannot show a dog a labelled dataset of correct behaviours. You can only reward what it does, and over many repetitions it works out which actions lead to treats.

The agent-environment loop

Every RL system runs the same cycle: the agent observes the current situation, chooses an action, and the environment responds with a new situation and a reward. That repeats until the episode ends.

The Gymnasium documentation, the most widely used interface for RL environments, expresses that loop in two calls.

• reset() starts a new episode and returns two values, an observation and an info dictionary.
• step(action) applies an action and returns five values: observation, reward, terminated, truncated, info.

The two end-of-episode flags matter, and Gymnasium keeps them separate on purpose. The flag terminated is true when the task itself ends, because it was completed or failed. The flag truncated is true when an outside limit ends the episode, such as a fixed number of timesteps.

The older Gym API returned four values, collapsing both cases into a single done flag. Gymnasium publishes a migration guide for that change. Check which version a tutorial targets before copying its loop, because the two signatures are not interchangeable.

The vocabulary you need

These six terms cover most of what you will read in RL papers and documentation.

• State: what the agent can observe about the environment right now.
• Action: a choice the agent can make in a state.
• Reward: the number the environment returns after an action. It is the only training signal.
• Policy: the agent's rule for choosing an action in a given state. This is what training produces.
• Value: the total future reward the agent expects from a state, or from a state and action pair, if it keeps following its policy.
• Episode: one run from a starting state to an ending state.

Most algorithms you meet are different ways of learning a good policy, and others learn a good value estimate that implies one.

Exploration vs. exploitation

An RL agent faces a constant trade-off. It can exploit what it already knows by repeating the best-known action, or explore by trying something new that might turn out better.

An agent that only exploits gets stuck with the first decent strategy it finds, while an agent that only explores never cashes in on what it learned.

Most methods mix the two. The simplest is epsilon-greedy: with a small probability pick a random action, and otherwise pick the best known one. The example below uses exactly that.

A runnable example: Q-learning on a grid

Q-learning is a classic RL algorithm that learns a value for every state and action pair, called a Q-value. This example uses no libraries, so you can run it as it stands.

The setup is a 4x4 grid where the agent starts at the top left and must reach the bottom right. Reaching the goal gives +1, and every other step costs 0.01, which pushes the agent toward short paths.

import random

# A 4x4 grid. The agent starts at (0, 0) and the goal is at (3, 3).
# Reward: +1 for reaching the goal, -0.01 for every other step.
SIZE = 4
GOAL = (3, 3)
ACTIONS = {"up": (-1, 0), "down": (1, 0), "left": (0, -1), "right": (0, 1)}
NAMES = list(ACTIONS)

def step(state, action):
    dr, dc = ACTIONS[action]
    r = min(max(state[0] + dr, 0), SIZE - 1)
    c = min(max(state[1] + dc, 0), SIZE - 1)
    nxt = (r, c)
    if nxt == GOAL:
        return nxt, 1.0, True
    return nxt, -0.01, False

random.seed(0)
Q = {(r, c): {a: 0.0 for a in NAMES} for r in range(SIZE) for c in range(SIZE)}
alpha, gamma, epsilon = 0.5, 0.95, 0.2

for episode in range(500):
    state, done = (0, 0), False
    while not done:
        if random.random() < epsilon:
            action = random.choice(NAMES)            # explore
        else:
            action = max(Q[state], key=Q[state].get)  # exploit
        nxt, reward, done = step(state, action)
        best_next = 0.0 if done else max(Q[nxt].values())
        Q[state][action] += alpha * (reward + gamma * best_next - Q[state][action])
        state = nxt

# Follow the learned greedy policy from the start.
state, path = (0, 0), [(0, 0)]
while state != GOAL and len(path) < 20:
    action = max(Q[state], key=Q[state].get)
    state, _, _ = step(state, action)
    path.append(state)
print("Greedy path:", path)
print("Steps to goal:", len(path) - 1)

Run with random.seed(0) on Python 3.13, the output is:

Greedy path: [(0, 0), (0, 1), (1, 1), (2, 1), (2, 2), (2, 3), (3, 3)]
Steps to goal: 6

Six steps is the shortest possible route on this grid, since reaching (3, 3) needs three moves down and three moves right. The exact path can differ with another seed because several shortest routes tie, but the step count should still be 6 once training has converged.

We ran this on Python 3.13.15 with seeds 1, 2, 3, 4, 5, 42 and 123, and every one reached the goal in 6 steps.

The heart of the algorithm is a single line, the update rule:

Q[state][action] += alpha * (reward + gamma * best_next - Q[state][action])

Read it as nudging the old estimate toward what just happened, where alpha is the learning rate and gamma discounts future reward so that sooner is worth more than later.

How RL differs from supervised learning

	Supervised learning	Reinforcement learning
Training signal	The correct answer for each example	A reward, often delayed, with no correct action given
Data	A fixed labelled dataset	Experience the agent generates by acting
Feedback timing	Immediate, per example	Often after many steps
Typical goal	Predict a label or value	Choose actions that maximise total reward
Main difficulty	Getting enough good labels	Credit assignment and exploration

The practical difference is that in RL the agent's own choices change what data it sees next. That feedback loop is powerful, and it is also why RL is harder to stabilise than supervised training.

Reinforcement learning and language models

RL is no longer only about games and robots. It is part of how modern language models are trained, in two main ways.

RLHF, learning from human preferences. In the InstructGPT work, Ouyang et al. first fine-tuned GPT-3 on human demonstrations. They then applied reinforcement learning from human feedback, using human rankings of model outputs as the signal.

Their headline result is worth stating precisely, because human evaluators preferred outputs from the 1.3B-parameter InstructGPT model to outputs from the 175B-parameter GPT-3, in the authors' words "despite having 100x fewer parameters". Here the reward is a learned stand-in for human preference. See the glossary entries on RLHF and the reward model for the pipeline in brief.

RL on verifiable tasks. The DeepSeek-R1 paper reports in its abstract that reasoning can be incentivised through pure reinforcement learning. It states this obviates the need for human-labelled reasoning trajectories.

The abstract reports superior performance on verifiable tasks such as mathematics, coding competitions and STEM fields, and describes emergent reasoning patterns including self-reflection, verification and dynamic strategy adaptation. When a program can check the answer, the reward is cheap and objective.

If you build with LLM APIs, you are usually using the output of RL rather than running RL yourself, but it still explains why models behave as they do, including why they can be agreeable in ways that reflect their reward signal.

How RL connects to world models

A world model is a learned predictor of what happens after an action, and RL supplies the other half: a way to choose between those actions.

The two combine in model-based RL. There the agent uses its world model to imagine outcomes and picks the action that scores best, rather than learning only by trial and error in the real environment.

That is the plan-by-prediction loop described in the guide to what world models are. Q-learning, as in the example above, is model-free: it never predicts the next state, and only learns from experience what each action is worth.

Model-based methods trade extra machinery for fewer real-world trials, which matters most when every trial is a physical robot.

Limits and failure modes

RL is powerful and also easy to get wrong. The paper Concrete Problems in AI Safety names several open problems that apply directly to RL systems.

• Reward hacking: the agent games the objective in ways the designer did not intend.
• Avoiding side effects: the agent changes its environment in unwanted ways while chasing reward.
• Safe exploration: the learning process itself causes harm before the agent knows better.
• Distributional shift: behaviour degrades when the situation differs from training.

Reward hacking is the one practitioners hit most, because the reward is only a proxy for what you want and a capable optimiser will find the gaps.

If you design a reward, test it by asking one question: what is the cheapest way to score high, and would you be happy with that behaviour?

Sources

Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed. (MIT Press, 2018). Standard framing of agent, environment, reward and policy, accessed 18 Oct 2026.

Gymnasium, Basic Usage. The reset and step loop, the five-value step return, and the terminated versus truncated distinction, accessed 18 Oct 2026.

Ouyang et al., Training language models to follow instructions with human feedback. The RLHF method and the 1.3B versus 175B preference result, accessed 18 Oct 2026.

DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Abstract-level claims on RL for reasoning, accessed 18 Oct 2026.

Amodei et al., Concrete Problems in AI Safety. Reward hacking, side effects, safe exploration and distributional shift, accessed 18 Oct 2026.

Methodology: definitions follow the Sutton and Barto framing. Every numeric claim about InstructGPT and DeepSeek-R1 is taken from the papers' own abstracts rather than secondary coverage. The Q-learning code was executed on Python 3.13.15 before publication, across eight seeds, and the output shown is the output it produced.