Landmark 3 stops to get here · leads to 4

Policy

The rule an RL agent uses to choose actions: a mapping from each state to an action, or to a probability distribution over actions.

Your route here

3 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Reinforcement Learning ✓ understood

    Learning through interaction with an environment, receiving rewards or penalties to learn optimal behavior policies.

  3. Agent ✓ understood

    In reinforcement learning, the learner and decision-maker that observes an environment, takes actions and adjusts its behaviour to maximize cumulative reward.

  4. Policy · you are here ✓ understood

Picture it

Observe state sWhat the agent sees right nowPolicy picks action aπ(a | s): a rule or a distributionEnvironment respondsThe world changesReward + next stateFeedback used to improve the…Policy π: state → action
  1. 01 Observe state s What the agent sees right now
  2. 02 Policy picks action a π(a | s): a rule or a distribution
  3. 03 Environment responds The world changes
  4. 04 Reward + next state Feedback used to improve the policy

↺ back to 01 · Policy π: state → action

Notice the policy sits at the decision point of every loop: given the state, it decides the action, and learning reshapes that choice.

In reinforcement learning, a policy is the agent’s way of choosing what to do. It takes the current state of the environment and returns an action, or a set of probabilities over actions. Everything an RL agent learns is ultimately stored as its policy: training exists to make it pick better actions.

The usual notation is π(a | s), read “the probability of taking action a in state s”. A chess policy maps a board position to a move. A robot’s policy maps joint angles and camera input to motor commands.

Two kinds

  • Deterministic: each state always gets the same action. Simple, but predictable, and it can’t explore on its own.
  • Stochastic: each state gets a probability distribution, and the action is sampled from it. This builds in exploration and is the natural fit when the best move really is to mix, as in rock-paper-scissors.

Where it sits in the loop

The diagram shows the policy at the decision point of every cycle. The agent observes a state, the policy picks an action, the environment responds, and a reward plus the next state come back. Learning then adjusts the policy so that actions leading to more total reward become more likely. The goal is not the biggest immediate reward but the biggest cumulative one, which is why a good policy will sometimes accept a small loss now for a larger gain later.

How policies are learned

  • Value-based: learn a value function that estimates how good each action is in each state, then act greedily with respect to it. The policy is implicit. Q-learning and DeepMind’s Atari-playing DQN work this way.
  • Policy-based: represent the policy directly, often as a neural network that outputs action probabilities, and nudge its parameters in the direction that raises expected reward. This is policy gradient.
  • Actor-critic: combine both, with a learned value estimate guiding the policy’s updates. PPO (2017) is a widely used example; it clips each update so the new policy can’t drift too far from the old one in a single step.

Policies in language models

In RLHF, the language model itself is the policy: the state is the conversation so far, and the action is the next token. OpenAI’s InstructGPT work fine-tuned such a policy with PPO against a learned reward model. The catch is the same as anywhere in RL: a policy optimizes exactly the reward it is given, so a flawed reward signal gets exploited rather than ignored.

Where it sits

Before this

Agent
Policy

Leads to

Policy Gradient Value Function Hierarchical RL Offline RL

Explore nearby

Agents & RL Value Function A function estimating expected cumulative reward from a state (state-value) or state-action pair (action-value/Q-value). Agents & RL Reward The single number an environment sends back after each action, telling a reinforcement learning agent how good that step was; the agent learns to maximize its long-run total. Agents & RL Environment In RL, the world the agent interacts with, providing states, accepting actions, and returning rewards. Agents & RL Policy Gradient RL methods that directly optimize the policy by computing gradients of expected reward with respect to policy parameters. Agents & RL Markov Decision Process A mathematical framework for modeling sequential decision-making with states, actions, rewards, and transition probabilities.

In the research

All papers →

4 papers that build on Policy .

Sources

  1. Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction . 2nd edition, MIT Press, 2018
  2. Schulman et al., "Proximal Policy Optimization Algorithms" . 2017
  3. Ouyang et al., "Training language models to follow instructions with human feedback" . 2022 (InstructGPT)