Agents & RL Jan 2016

Mastering the Game of Go with Deep Neural Networks and Tree Search

David Silver et al. · Nature

doi:10.1038/nature16961

In short

AlphaGo combines a policy network that suggests moves and a value network that judges positions with Monte Carlo tree search. Trained on expert games and then by playing itself, it became the first program to beat a professional Go player on a full board.

Why it matters

It proved that learned intuition plus search can master a problem long thought a decade away.

Read first

The 4 Field Guide ideas this paper leans on.

Starting from scratch? The full route 8 ideas · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Reinforcement Learning · read first ✓ understood

    Learning through interaction with an environment, receiving rewards or penalties to learn optimal behavior policies.

  3. Agent ✓ understood

    In reinforcement learning, the learner and decision-maker that observes an environment, takes actions and adjusts its behaviour to maximize cumulative reward.

  4. Policy · read first ✓ understood

    The rule an RL agent uses to choose actions: a mapping from each state to an action, or to a probability distribution over actions.

  5. Reward ✓ understood

    The single number an environment sends back after each action, telling a reinforcement learning agent how good that step was; the agent learns to maximize its long-run total.

  6. Value Function · read first ✓ understood

    A function estimating expected cumulative reward from a state (state-value) or state-action pair (action-value/Q-value).

  7. Exploration vs Exploitation ✓ understood

    The RL dilemma of trying new actions (exploration) versus using known good actions (exploitation) to maximize reward.

  8. Monte Carlo Tree Search · read first ✓ understood

    A search algorithm combining tree search with random sampling, used in game-playing AIs.

Nearby papers

Summary in our own words; read the paper for the details. ← All papers