Neural Networks Dec 2012

ImageNet Classification with Deep Convolutional Neural Networks

Alex Krizhevsky et al. · NeurIPS 2012 (reprinted in Communications of the ACM, 2017)

doi:10.1145/3065386

In short

A large convolutional network, trained on two GPUs with ReLU activations and dropout, won the 2012 ImageNet challenge by a wide margin over every hand-engineered entry. The paper details the architecture and the tricks that made training at that scale work.

Why it matters

This is the moment deep learning went mainstream: GPUs plus data plus depth beat decades of feature engineering.

Read first

The 4 Field Guide ideas this paper leans on.

Starting from scratch? The full route 15 ideas · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  3. Convolution ✓ understood

    A mathematical operation that applies filters/kernels to input data to extract features like edges, textures, and patterns.

  4. Convolutional Neural Network · read first ✓ understood

    A neural network that scans images with small learned filters, reusing the same weights at every position to build up from edges to whole objects.

  5. GPU · read first ✓ understood

    Graphics Processing Unit - hardware accelerator with thousands of cores, essential for parallel computation in deep learning.

  6. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  7. Training Data ✓ understood

    The examples a model learns its weights from, kept separate from the validation and test data used to check how well it generalizes.

  8. Overfitting ✓ understood

    When a model fits its training data too closely, noise included, so it scores well on examples it has seen and poorly on new ones.

  9. Dropout · read first ✓ understood

    A regularization technique that randomly switches off units during training so the network can't lean on any one of them, which reduces overfitting.

  10. Supervised Learning ✓ understood

    Learning from examples paired with the correct answer, so a model can predict answers for new inputs it hasn't seen.

  11. Classification ✓ understood

    A supervised learning task where the model assigns each input to one of a fixed set of categories, such as spam or not spam.

  12. Deep Learning ✓ understood

    A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.

  13. Computer Vision ✓ understood

    The field of AI that gets computers to extract meaning from images and video: what is in them, where it is, and how it moves.

  14. Image Classification ✓ understood

    Assigning one label from a fixed set of categories to a whole image, such as "cat" or "defective part"; a core computer vision task.

  15. ImageNet · read first ✓ understood

    A large-scale dataset of 14M images in 20K categories, historically used as the benchmark for image classification models.

Nearby papers

Summary in our own words; read the paper for the details. ← All papers