Agents & RL Dec 2013

Playing Atari with Deep Reinforcement Learning

Volodymyr Mnih et al. · NeurIPS 2013 Deep Learning Workshop

arXiv:1312.5602

In short

A convolutional network trained with a variant of Q-learning learns to play Atari games straight from screen pixels and the score. The same network and settings worked across games, beating previous methods on six of seven and a human expert on three.

Why it matters

It started deep reinforcement learning: one learner, raw perception, many tasks.

Read first

The 4 Field Guide ideas this paper leans on.

Agents & RL · Landmark Reinforcement Learning Learning through interaction with an environment, receiving rewards or penalties to learn optimal behavior policies. Neural Networks · Landmark Convolutional Neural Network A neural network that scans images with small learned filters, reusing the same weights at every position to build up from edges to whole objects. Agents & RL · Standard Q-Learning A model-free RL algorithm that learns action-value functions (Q-values) to determine optimal actions in each state. Agents & RL · Standard Deep Q-Network Combining Q-learning with deep neural networks to handle high-dimensional state spaces, enabling RL for complex tasks like Atari games.

Starting from scratch? The full route 11 ideas · basics first

Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
Reinforcement Learning · read first ✓ understood
Learning through interaction with an environment, receiving rewards or penalties to learn optimal behavior policies.
Neural Network ✓ understood
A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.
Convolution ✓ understood
A mathematical operation that applies filters/kernels to input data to extract features like edges, textures, and patterns.
Convolutional Neural Network · read first ✓ understood
A neural network that scans images with small learned filters, reusing the same weights at every position to build up from edges to whole objects.
Reward ✓ understood
A scalar feedback signal indicating how good an action was, used to train reinforcement learning agents.
Agent ✓ understood
In RL, the learner or decision-maker that takes actions in an environment to maximize cumulative reward.
Policy ✓ understood
A strategy or mapping from states to actions that defines the agent's behavior in reinforcement learning.
Value Function ✓ understood
A function estimating expected cumulative reward from a state (state-value) or state-action pair (action-value/Q-value).
Q-Learning · read first ✓ understood
A model-free RL algorithm that learns action-value functions (Q-values) to determine optimal actions in each state.
Deep Q-Network · read first ✓ understood
Combining Q-learning with deep neural networks to handle high-dimensional state spaces, enabling RL for complex tasks like Atari games.

Nearby papers

Canon · 1998 Gradient-Based Learning Applied to Document Recognition It is the blueprint for the convolutional network, the architecture that later cracked computer vision. Canon · 2012 ImageNet Classification with Deep Convolutional Neural Networks This is the moment deep learning went mainstream: GPUs plus data plus depth beat decades of feature engineering. Canon · 2015 Deep Residual Learning for Image Recognition Residual connections are now in nearly every deep network, including every transformer. Canon · 2016 Mastering the Game of Go with Deep Neural Networks and Tree Search It proved that learned intuition plus search can master a problem long thought a decade away.