Standard 8 stops to get here

Bellman Equation

The recursive rule at the heart of reinforcement learning: a state's value equals the reward you get now plus the discounted value of where you land next.

Your route here

8 stops · basics first

Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
Reinforcement Learning ✓ understood
Learning through interaction with an environment, receiving rewards or penalties to learn optimal behavior policies.
Reward ✓ understood
The single number an environment sends back after each action, telling a reinforcement learning agent how good that step was; the agent learns to maximize its long-run total.
Agent ✓ understood
In reinforcement learning, the learner and decision-maker that observes an environment, takes actions and adjusts its behaviour to maximize cumulative reward.
Environment ✓ understood
In RL, the world the agent interacts with, providing states, accepting actions, and returning rewards.
Markov Decision Process ✓ understood
A mathematical framework for modeling sequential decision-making with states, actions, rewards, and transition probabilities.
Policy ✓ understood
The rule an RL agent uses to choose actions: a mapping from each state to an action, or to a probability distribution over actions.
Value Function ✓ understood
A function estimating expected cumulative reward from a state (state-value) or state-action pair (action-value/Q-value).
Bellman Equation · you are here ✓ understood

Picture it

01 Value of state s What we want to know
02 Reward for next step What the next action pays right away
03 + γ × value of next state The rest of the future, discounted

Notice the recursion: a state's value is defined through the value of the state after it, so one rule links every state to its neighbours.

The Bellman equation says that the value of being somewhere is what you collect on the next step plus the (discounted) value of wherever that step takes you. It turns a question about the whole future into a question about one step and its successor.

In a Markov decision process, the value function of a policy gives the expected total discounted reward from each state. Written out, the value of state s averages, over the actions the policy might take and the states they might lead to, the immediate reward plus γ times the value of the next state. The discount γ, between 0 and 1, makes rewards further in the future count for less. Sutton and Barto note that a policy’s value function is the unique solution to its Bellman equation.

Two versions

Bellman expectation equation: describes the value of following a fixed policy. It answers “how good is this plan?”
Bellman optimality equation: replaces the average over the policy’s actions with a maximum over actions. Its solution is the value of acting optimally, and once you have it, acting greedily with respect to it is an optimal policy.

Why it matters

The equation is the common thread through most of reinforcement learning. Dynamic programming, the family of methods named by Richard Bellman in the 1950s, solves it directly when the environment’s dynamics are known. Q-learning applies the optimality version to action values, nudging each estimate toward the observed reward plus the best estimated value of the next state. DQN does the same with a neural network standing in for the table. Each of these updates is a “Bellman backup”: use one step of real experience to make the value estimate more consistent with itself.

Where it sits

Before this

Markov Decision Process Value Function

Bellman Equation

Leads to

Nothing yet: a destination in its own right.

Explore nearby

Agents & RL Q-Learning A model-free RL algorithm that learns action-value functions (Q-values) to determine optimal actions in each state. Agents & RL Deep Q-Network Combining Q-learning with deep neural networks to handle high-dimensional state spaces, enabling RL for complex tasks like Atari games. Agents & RL Actor-Critic RL architecture with two components: an actor (policy) that selects actions and a critic (value function) that evaluates them. Agents & RL Model-Based RL Reinforcement learning using learned environment models for planning and improving sample efficiency. Agents & RL Policy The rule an RL agent uses to choose actions: a mapping from each state to an action, or to a probability distribution over actions. Agents & RL Reward The single number an environment sends back after each action, telling a reinforcement learning agent how good that step was; the agent learns to maximize its long-run total.

Sources

Sutton and Barto, Reinforcement Learning: An Introduction (2nd ed.) . MIT Press, 2018; chapter 3 (Bellman equations), chapter 4 (dynamic programming), section 1.7 (history, including Bellman's 1957 work)