A reward is the feedback signal in reinforcement learning: after every action, the environment hands the agent a single number. Positive means that step helped, negative means it hurt, and zero means nothing much happened. There are no labelled right answers, only this running score.
The idea behind it is known as the reward hypothesis: any goal can be expressed as maximizing the expected cumulative sum of one scalar signal. Winning a game, balancing a pole, keeping a user engaged: each gets boiled down to “more reward is better.”
From reward to return
The agent doesn’t try to grab the biggest reward right now. It tries to maximize the return, the sum of all the rewards still to come. That’s why the diagram adds them up before anything gets updated. A discount rate γ between 0 and 1 makes a reward k steps away worth γ^k of the same reward received now. With γ at 0 the agent is myopic and only cares about the next step; close to 1, it plans far ahead.
Because the payoff can arrive long after the move that earned it, the agent has to work out which earlier actions deserve the credit. A value function estimates the return expected from a state, and the policy shifts toward actions that lead to higher values.
Designing the signal
Choosing the reward is where the human’s intent enters the system. The standard advice is to reward the outcome you want, not the steps you think lead to it. Sutton and Barto’s example: reward a chess agent for winning, not for capturing pieces, or it may learn to capture pieces and lose.
Sometimes no one can write the reward down. Inverse reinforcement learning infers one from expert demonstrations. RLHF trains a reward model on human rankings of model outputs, then optimizes a language model against it; that’s how InstructGPT was tuned.
The catch
An agent optimizes the reward you wrote, not the one you meant. This is called reward hacking. A 2016 paper on AI safety gives the example of a cleaning robot rewarded for not seeing any mess. It might learn to disable its own vision rather than clean. A learned reward model can be gamed the same way, which is why InstructGPT’s training added a penalty for drifting too far from the model it started from.