Mastering the Game of Go with Deep Neural Networks and Tree Search
David Silver et al. · Nature
doi:10.1038/nature16961
In short
AlphaGo combines a policy network that suggests moves and a value network that judges positions with Monte Carlo tree search. Trained on expert games and then by playing itself, it became the first program to beat a professional Go player on a full board.
Why it matters
It proved that learned intuition plus search can master a problem long thought a decade away.
Read first
The 4 Field Guide ideas this paper leans on.
Starting from scratch? The full route 8 ideas · basics first
- Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
- Reinforcement Learning · read first ✓ understood
Learning through interaction with an environment, receiving rewards or penalties to learn optimal behavior policies.
- Agent ✓ understood
In reinforcement learning, the learner and decision-maker that observes an environment, takes actions and adjusts its behaviour to maximize cumulative reward.
- Policy · read first ✓ understood
The rule an RL agent uses to choose actions: a mapping from each state to an action, or to a probability distribution over actions.
- Reward ✓ understood
The single number an environment sends back after each action, telling a reinforcement learning agent how good that step was; the agent learns to maximize its long-run total.
- Value Function · read first ✓ understood
A function estimating expected cumulative reward from a state (state-value) or state-action pair (action-value/Q-value).
- Exploration vs Exploitation ✓ understood
The RL dilemma of trying new actions (exploration) versus using known good actions (exploitation) to maximize reward.
- Monte Carlo Tree Search · read first ✓ understood
A search algorithm combining tree search with random sampling, used in game-playing AIs.