Landmark 2 stops to get here · leads to 5

Deep Learning

A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.

Your route here

2 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  3. Deep Learning · you are here ✓ understood

Picture it

  1. Prediction The output label
  2. Whole objects Faces, cars
  3. Parts Eyes, wheels, windows
  4. Textures and corners
  5. Edges Early layers
  6. Raw pixels The input
Reading upward, each layer builds more abstract features from the one below, learned from data rather than hand-made.

Deep learning is machine learning with neural networks that have many layers. “Deep” just means the number of layers stacked between input and output: a handful in early networks, hundreds in some modern ones. It is the approach behind today’s language, image, speech and video models.

What sets it apart is who designs the features. In classic machine learning, people decided what the model should look at: word counts for text, edge detectors for images, hand-built measurements for audio. Deep learning feeds in raw data, pixels or tokens, and lets the layers learn their own features. That is representation learning, and the depth is what makes it work.

Why depth helps

The diagram shows the classic picture from image networks. Reading upward, early layers respond to simple patterns such as edges, middle layers combine those into textures and parts, and later layers respond to whole objects. Nobody programs this hierarchy. It emerges from training, because building complex features out of simpler ones is an efficient way to fit the data. The standard textbook frames deep learning as exactly this: a hierarchy of concepts, each defined through simpler ones.

Every layer is trained together, end to end. Backpropagation sends the error signal from the output back through every layer, and gradient descent adjusts all the weights at once.

Why it took off around 2012

The core ideas are decades old. Three things came together to make them practical:

  • Data: large labeled sets like ImageNet, and later the text of the web.
  • Compute: GPUs, which do the matrix arithmetic of neural networks far faster than general-purpose processors.
  • Training fixes: ReLU activations, dropout and better initialization, which let deeper networks train reliably.

The turning point was AlexNet in 2012. A deep convolutional network trained on two GPUs won that year’s ImageNet competition with a top-5 error of 15.3%, against 26.2% for the next-best entry. Deep networks then spread to speech, translation and games, and the transformer extended the same recipe to language at enormous scale.

The catch

Deep models are hungry. They generally need a lot of data and compute, and on small or tabular datasets simpler methods, such as gradient-boosted trees, can do as well. They are also hard to inspect: the learned features are spread across millions or billions of weights, so explaining why a model made a particular decision remains an open research problem.

Where it sits

Explore nearby

Sources

  1. LeCun, Bengio and Hinton, "Deep learning" . Nature, 2015
  2. Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning . MIT Press, 2016; chapter 1, Introduction
  3. Krizhevsky, Sutskever and Hinton, "ImageNet Classification with Deep Convolutional Neural Networks" . NeurIPS 2012